AWS Cloudformation - mount to existing file system - amazon-web-services

Currently, I have a json that will create a EFS in an auto scaling group. However, how can I make it so it mounts an existing EFS that is previously created (so i can pre-load data)
this is the current set up
"FileSystem": {
"Type": "AWS::EFS::FileSystem",
"Properties": {
"PerformanceMode": "generalPurpose",
"FileSystemTags": [
{
"Key": "Name",
"Value": { "Ref" : "VolumeName" }
}
]
}
},
"MountTarget": {
"Type": "AWS::EFS::MountTarget",
"Properties": {
"FileSystemId": { "Ref": "FileSystem" },
"SubnetId": { "Ref": "Subnet" },
"SecurityGroups": [ { "Ref": "MountTargetSecurityGroup" } ]
}
},

As per #MaiKaY Suggested, Just Pass the FileSystemId as a Parameter.
here is the mounting Script
container_commands:
1chown:
command: "chown webapp:webapp /wpfiles"
2create:
command: "sudo -u webapp mkdir -p wp-content/uploads"
3link:
command: "sudo -u webapp ln -s /wpfiles wp-content/uploads"
option_settings:
aws:elasticbeanstalk:application:environment:
FILE_SYSTEM_ID: 'FileSystemId' #Provide your FileSystemId
MOUNT_DIRECTORY: '/wpfiles'
REGION: '`{"Ref": "AWS::Region"}`'
packages:
yum:
nfs-utils: []
jq: []
commands:
01_mount:
command: "/tmp/mount-efs.sh"
files:
"/tmp/mount-efs.sh":
mode: "000755"
content : |
#!/bin/bash
EFS_REGION=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.REGION')
EFS_MOUNT_DIR=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MOUNT_DIRECTORY')
EFS_FILE_SYSTEM_ID=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.FILE_SYSTEM_ID')
echo "Mounting EFS filesystem ${EFS_DNS_NAME} to directory ${EFS_MOUNT_DIR} ..."
echo 'Stopping NFS ID Mapper...'
service rpcidmapd status &> /dev/null
if [ $? -ne 0 ] ; then
echo 'rpc.idmapd is already stopped!'
else
service rpcidmapd stop
if [ $? -ne 0 ] ; then
echo 'ERROR: Failed to stop NFS ID Mapper!'
exit 1
fi
fi
echo 'Checking if EFS mount directory exists...'
if [ ! -d ${EFS_MOUNT_DIR} ]; then
echo "Creating directory ${EFS_MOUNT_DIR} ..."
mkdir -p ${EFS_MOUNT_DIR}
if [ $? -ne 0 ]; then
echo 'ERROR: Directory creation failed!'
exit 1
fi
else
echo "Directory ${EFS_MOUNT_DIR} already exists!"
fi
mountpoint -q ${EFS_MOUNT_DIR}
if [ $? -ne 0 ]; then
echo "mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ${EFS_FILE_SYSTEM_ID}:/ ${EFS_MOUNT_DIR}"
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ${EFS_FILE_SYSTEM_ID}:/ ${EFS_MOUNT_DIR}
if [ $? -ne 0 ] ; then
echo 'ERROR: Mount command failed!'
exit 1
fi
chmod 777 ${EFS_MOUNT_DIR}
runuser -l ec2-user -c "touch ${EFS_MOUNT_DIR}/it_works"
if [[ $? -ne 0 ]]; then
echo 'ERROR: Permission Error!'
exit 1
else
runuser -l ec2-user -c "rm -f ${EFS_MOUNT_DIR}/it_works"
fi
else
echo "Directory ${EFS_MOUNT_DIR} is already a valid mountpoint!"
fi
echo 'EFS mount complete.'

Related

Script exited with non-zero exit status: 100.Allowed exit codes are: [0] : packer error [duplicate]

I have a shell provisioner in packer connected to a box with user vagrant
{
"environment_vars": [
"HOME_DIR=/home/vagrant"
],
"expect_disconnect": true,
"scripts": [
"scripts/foo.sh"
],
"type": "shell"
}
where the content of the script is:
whoami
sudo su
whoami
and the output strangely remains:
==> virtualbox-ovf: Provisioning with shell script: scripts/configureProxies.sh
virtualbox-ovf: vagrant
virtualbox-ovf: vagrant
why cant I switch to the root user?
How can I execute statements as root?
Note, I do not want to quote all statements like sudo "statement |foo" but rather globally switch user like demonstrated with sudo su
You should override the execute_command. Example:
"provisioners": [
{
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E sh -eux '{{.Path}}'",
"scripts": [
"scripts/foo.sh"
],
"type": "shell"
}
],
There is another solution with simpler usage of 2 provisioner together.
Packer's shell provisioner can run the bash with sudo privileges. First you need copy your script file from local machine to remote with file provisioner, then run it with shell provisioner.
packer.json
{
"vars": [...],
"builders": [
{
# ...
"ssh_username": "<some_user_other_than_root_with_passwordless_sudo>",
}
],
"provisioners": [
{
"type": "file",
"source": "scripts/foo.sh",
"destination": "~/shell.tmp.sh"
},
{
"type": "shell",
"inline": ["sudo bash ~/shell.tmp.sh"]
}
]
}
foo.sh
# ...
whoami
sudo su root
whoami
# ...
output
<some_user_other_than_root_with_passwordless_sudo>
root
After provisioner complete its task, you can delete the file with shell provisioner.
packer.json updated
{
"type": "shell",
"inline": ["sudo bash ~/shell.tmp.sh", "rm ~/shell.tmp.sh"]
}
one possible answer seems to be:
https://unix.stackexchange.com/questions/70859/why-doesnt-sudo-su-in-a-shell-script-run-the-rest-of-the-script-as-root
sudo su <<HERE
ls /root
whoami
HERE
maybe there is a better answer?
Assuming that the shell provisioner you are using is a bash script, you can add my technique to your script.
function if_not_root_rerun_as_root(){
install_self
if [[ "$(id -u)" -ne 0 ]]; then
run_as_root_keeping_exports "$0" "$#"
exit $?
fi
}
function run_as_root_keeping_exports(){
eval sudo $(for x in $_EXPORTS; do printf '%s=%q ' "$x" "${!x}"; done;) "$#"
}
export EXPORTS="PACKER_BUILDER_TYPE PACKER_BUILD_NAME"
if_not_root_rerun_as_root "$#"
There is a pretty good explanation of "$#" here on StackOverflow.

Amazon's AWS ElasticBeanstalk Let's Encrypt CertBot

How can I run and automated scalable version of Let's encrypt Certbot on Elastic beanstalk ? Haven't found an answer on the web so I put together this .ebextension. I hope you find it useful.
Don't use as is, check and modify according to your needs.
Requirements:
ElasticBeanstalk Environmnet
Ec2-elastic-beanstalk-service-role with access to Route53 Resource
TCP Loadbalancer in front of EC2 instances
Deployment policy rolling (one instance at a the time)
NFS mount
It's a single .ebextension:
jq package is needed:
packages:
yum:
jq: []
Creating jsons:
files:
"/usr/local/bin/certbot/add_json.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
#Create TXT record JSON
cat > $1_TXT.json << END
{
"Comment": "TXT Verification for CertBOT",
"Changes": [
{
"Action": "$1",
"ResourceRecordSet": {
"Name": "_acme-challenge.$CERTBOT_DOMAIN",
"Type": "TXT",
"TTL": 300,
"ResourceRecords": [
{
"Value": "\"$CERTBOT_VALIDATION\""
}
]
}
}
]
}
END
Removing hook to Route53:
"/usr/local/bin/certbot/remove_txt_hook.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
PWD=`pwd`
APEX_DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')
ZONE_ID=`aws route53 list-hosted-zones --output text | awk '$4 ~ /^ *'$APEX_DOMAIN'/''{print $3}' | sed 's:.*/::'`
aws route53 change-resource-record-sets \
--hosted-zone-id $ZONE_ID --change-batch file://$PWD/DELETE_TXT.json
rm CREATE_TXT.json
rm DELETE_TXT.json
Adding hook to Route53:
"/usr/local/bin/certbot/add_txt_hook.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
PWD=`pwd`
APEX_DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')
ZONE_ID=`aws route53 list-hosted-zones --output text | awk '$4 ~ /^ *'$APEX_DOMAIN'/''{print $3}' | sed 's:.*/::'`
./add_json.sh CREATE
./add_json.sh DELETE
aws route53 change-resource-record-sets \
--hosted-zone-id $ZONE_ID --change-batch file://$PWD/CREATE_TXT.json
sleep 30
Deploying:
"/usr/local/bin/certbot/start_process.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
MY_DOMAIN=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MY_DOMAIN')
EFS_NAME=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.EFS_NAME')
PWD=`pwd`
if [ "$MY_DOMAIN" = example.com ]; then
if [ ! -d /etc/letsencrypt ]; then
mkdir -p /etc/letsencrypt
fi
if ! grep -qs ' /etc/letsencrypt ' /proc/mounts; then
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_NAME:/.certificates/.letsencrypt /etc/letsencrypt || exit
fi
if [ ! -f certbot-auto ]; then
yum -y install mod24_ssl augeas-libs libffi-devel python27-tools system-rpm-config
wget https://dl.eff.org/certbot-auto
chmod 550 certbot-auto
fi
if [ ! -f /etc/letsencrypt/live/$MY_DOMAIN/fullchain.pem ]; then
./certbot-auto certonly --debug -n --no-bootstrap --email <your e-mail> --agree-tos --manual-public-ip-logging-ok --manual --preferred-challenges=dns --manual-auth-hook $PWD/add_txt_hook.sh --manual-cleanup-hook $PWD/remove_txt_hook.sh -d $MY_DOMAIN
fi
echo "00 15 * * SUN root cd /usr/local/bin/certbot && ./renewal.sh >> certbot.log 2>&1" > /etc/cron.d/cron_certbot
fi
Renewing:
"/usr/local/bin/certbot/renewal.sh":
mode: "000550"
owner: root
group: root
content: |
##!/bin/bash
MY_DOMAIN=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MY_DOMAIN')
PWD=`pwd`
ENV_ID=`{"Ref": "AWSEBEnvironmentId" }`
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
TODAY=`date +%Y-%m-%d`
STATUS=`aws elasticbeanstalk describe-environments --environment-ids $ENV_ID --region $REGION | awk '/"Status"/ {print substr($2, 1, length($2)-1)}' | sed 's/\"//g'`
while [ "$STATUS" != "Ready" ]; do
STATUS=`aws elasticbeanstalk describe-environments --environment-ids $ENV_ID --region $REGION | awk '/"Status"/ {print substr($2, 1, length($2)-1)}' | sed 's/\"//g'`
sleep 10
done
if ! /usr/local/bin/certbot/one_instance.sh; then
i="0"
while [ "$i" -lt 180 ] && [ ! -f /etc/letsencrypt/renewed_$TODAY ]; do
i=$[$i+1]
sleep 2
done
if [ ! -f /etc/letsencrypt/renewed_$TODAY ]; then
exit
else
/etc/init.d/httpd graceful; exit
fi
fi
./certbot-auto renew --debug --no-bootstrap --renew-hook "/etc/init.d/httpd graceful; touch /etc/letsencrypt/renewed_$TODAY; find /etc/letsencrypt/ -type f -name 'renewed_*' -mtime +0 -exec rm {} \;"
Only on one instance:
"/usr/local/bin/certbot/one_instance.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output text | awk '/aws:autoscaling:groupName/ {print $5}'`
SOLO_I=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output text | awk '/InService/ {print $4}' | sort | head -1`
[ "$SOLO_I" = "$INSTANCE_ID" ]
Executing:
commands:
01_start_certbot_deploy:
command: "/usr/local/bin/certbot/start_process.sh &>> certbot.log"
cwd: "/usr/local/bin/certbot"
02_delete_bak_files:
command: "rm -f *.bak"
cwd: "/usr/local/bin/certbot"
Let me know if you have some remarks or improvement suggestions.
Thanks!

How to install supervisor in Elastic Beanstalk through .ebextension?

May I know how can I install supervisor into Elastic Beanstalk through .ebextension? And how can I execute supervisor command through .ebextension?
Supervisorctl and supervisord are already present on elasticbeanstalk instances in the /usr/local/bin directory. You can use ebextensions to load a supervisor config file and run supervisor in daemon mode.
In your .ebextensions folder create a file 002_supervisor.config.
This file does 3 things:
Creates a supervisor.conf file in /usr/local/etc on your elastic beanstalk instance.
Creates an init.d script so that supervisor will be ran as a daemon at system start
Runs restart on supervisor when the application is deployed
files:
/usr/local/etc/supervisord.conf:
mode: "000755"
owner: root
group: root
content: |
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /usr/local/etc/*.conf
[inet_http_server]
port = 127.0.0.1:9001
/etc/init.d/supervisord:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
# Source function library
. /etc/rc.d/init.d/functions
# Source system settings
if [ -f /etc/sysconfig/supervisord ]; then
. /etc/sysconfig/supervisord
fi
# Path to the supervisorctl script, server binary,
# and short-form for messages.
supervisorctl=/usr/local/bin/supervisorctl
supervisord=${SUPERVISORD-/usr/local/bin/supervisord}
prog=supervisord
pidfile=${PIDFILE-/tmp/supervisord.pid}
lockfile=${LOCKFILE-/var/lock/subsys/supervisord}
STOP_TIMEOUT=${STOP_TIMEOUT-60}
OPTIONS="${OPTIONS--c /usr/local/etc/supervisord.conf}"
RETVAL=0
start() {
echo -n $"Starting $prog: "
daemon --pidfile=${pidfile} $supervisord $OPTIONS
RETVAL=$?
echo
if [ $RETVAL -eq 0 ]; then
touch ${lockfile}
$supervisorctl $OPTIONS status
fi
return $RETVAL
}
stop() {
echo -n $"Stopping $prog: "
killproc -p ${pidfile} -d ${STOP_TIMEOUT} $supervisord
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -rf ${lockfile} ${pidfile}
}
reload() {
echo -n $"Reloading $prog: "
LSB=1 killproc -p $pidfile $supervisord -HUP
RETVAL=$?
echo
if [ $RETVAL -eq 7 ]; then
failure $"$prog reload"
else
$supervisorctl $OPTIONS status
fi
}
restart() {
stop
start
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p ${pidfile} $supervisord
RETVAL=$?
[ $RETVAL -eq 0 ] && $supervisorctl $OPTIONS status
;;
restart)
restart
;;
condrestart|try-restart)
if status -p ${pidfile} $supervisord >&/dev/null; then
stop
start
fi
;;
force-reload|reload)
reload
;;
*)
echo $"Usage: $prog {start|stop|restart|condrestart|try-restart|force-reload|reload}"
RETVAL=2
esac
exit $RETVAL
commands:
01_start_supervisor:
command: '/etc/init.d/supervisord restart'
leader_only: true
Hope this helps!

'VBoxManage guestcontrol' to run shell script on guest

I have VirtualBox VM that runs a server that can be accessed via localhost and forwarded ports.
I need to run some shells scripts and implement some business logic based on the results.
I tried following command as example:
VBoxManage guestcontrol <UUID> exec --image /bin/sh --username <su username> --password <su password> --wait-exit --wait-stdout --wait-stderr -- "[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed""
but I'm getting the error:
/bin/sh: [ -d <server_folder> ] && echo : No such file or directory
What is wrong with the syntax above?
First make sure that VBoxManage.exe is in your path!
Secondly you have to be carefull with your quotations. You used:
"[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed""
you have to use singel quotaions for the outermost quotation:
'[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed"'
Finally you have to add a -c in front of your arguments (to call /bin/sh -c '...').
The complete command:
VBoxManage guestcontrol <UUID> exec --image /bin/sh --username <su username> --password <su password> --wait-exit --wait-stdout --wait-stderr -- -c '[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed"'

How can I compilie openexr with mingw

I am trying to compile openexr with mingw on windows7. I am gettng this error:
main.cpp: In function 'int main(int, char**)':
main.cpp:213:28: error: 'strcmp' was not declared in this scope
make[1]: *** [main.o] Error 1
make[1]: Leaving directory `/d/openexr/build/openexr-1.6.1/exrmaketiled'
make: *** [all-recursive] Error 1
Why am I getting this error and how can I slove it?
I am new in mingw and openexr, so I may did something silly!
edit1:
I am using this script to make it:
!/bin/sh
# modified synfig build file, taken from
# http://www.synfig.org/Windows_build_instructions
echo "Making OpenEXR..."
# Including configuration
if [ -r "./make_openexr.conf" ]; then
. ./make_openexr.conf
else
echo "No config file (./make_openexr.conf) found."
exit 1
fi
CURRENT_DIR=`pwd`
echo "Cleanup directories"
rm -rf ${BUILD_DIR}/zlib-${ZLIB_VERSION}
[ $? -eq 0 ] || exit 1
rm -rf ${BUILD_DIR}/ilmbase-${ILMBASE_VERSION}
[ $? -eq 0 ] || exit 1
rm -rf ${BUILD_DIR}/openexr-${OPENEXR_VERSION}
[ $? -eq 0 ] || exit 1
echo "Preparing sources"
echo PATH is $PATH
echo tar -xzf ${ZLIB_SRC} -C ${BUILD_DIR}
tar -xzf ${ZLIB_SRC} -C ${BUILD_DIR}
[ $? -eq 0 ] || exit 1
echo tar -xzf ${ILMBASE_SRC} -C ${BUILD_DIR}
tar -xzf ${ILMBASE_SRC} -C ${BUILD_DIR}
[ $? -eq 0 ] || exit 1
echo tar -xzf ${OPENEXR_SRC} -C ${BUILD_DIR}
tar -xzf ${OPENEXR_SRC} -C ${BUILD_DIR}
[ $? -eq 0 ] || exit 1
echo "Configuring zlib"
cd ${BUILD_DIR}/zlib-${ZLIB_VERSION}
[ $? -eq 0 ] || exit 1
./configure --prefix=${TEMP_INSTALL}
[ $? -eq 0 ] || exit 1
echo "Making"
make
[ $? -eq 0 ] || exit 1
make install
[ $? -eq 0 ] || exit 1
echo "zlib Done"
echo "Applying patches for ILMBase... "
cd ${BUILD_DIR}/ilmbase-${ILMBASE_VERSION}
[ $? -eq 0 ] || exit 1
for SFILE in ${ILMBASE_PATCHES[#]}
do
patch -p1 <${PATCHES_DIR}/${SFILE}
[ $? -eq 0 ] || exit 1
done
echo "Configuring ILMbase"
[ $? -eq 0 ] || exit 1
./configure --host=${MINGW_HOST} --prefix=${TEMP_INSTALL} \
--disable-static --disable-threading --disable-posix-sem
[ $? -eq 0 ] || exit 1
echo "Making"
make
[ $? -eq 0 ] || exit 1
make install
[ $? -eq 0 ] || exit 1
echo "ILMBase Done"
echo "Applying patches for OpenEXR... "
cd ${BUILD_DIR}/openexr-${OPENEXR_VERSION}
[ $? -eq 0 ] || exit 1
for SFILE in ${OPENEXR_PATCHES[#]}
do
patch -p1 <${PATCHES_DIR}/${SFILE}
[ $? -eq 0 ] || exit 1
done
# Now the temp/openexr dir exists, we can add it to paths
if [ -d "${TEMP_INSTALL}" ]; then
PATH="${TEMP_INSTALL}/bin:${PATH}"
PKG_CONFIG_PATH="${TEMP_INSTALL}/lib/pkgconfig:${PKG_CONFIG_PATH}"
fi
# set inc/lib path (for zlib)
CXXFLAGS+=" -I${TEMP_INSTALL}/include"
LDFLAGS+=" -L${TEMP_INSTALL}/lib"
export PATH
export PKG_CONFIG_PATH
export CXXFLAGS
export LDFLAGS
echo "Configuring OpenEXR"
[ $? -eq 0 ] || exit 1
./configure --host=${MINGW_HOST} --prefix=${TEMP_INSTALL} \
--disable-static --disable-threading --disable-posix-sem --disable-ilmbasetest
[ $? -eq 0 ] || exit 1
echo "separately build b44ExpLogTable as the openexr script doesn't work under msys"
cd ${BUILD_DIR}/openexr-${OPENEXR_VERSION}/IlmImf
[ $? -eq 0 ] || exit 1
mingw32-g++ --verbose -g -O2 -I${TEMP_INSTALL}/include/OpenEXR -L${TEMP_INSTALL}/lib b44ExpLogTable.cpp -lHalf -o b44ExpLogTable
[ $? -eq 0 ] || exit 1
cd ..
[ $? -eq 0 ] || exit 1
echo "Making"
make
[ $? -eq 0 ] || exit 1
make install
[ $? -eq 0 ] || exit 1
cd ${CURRENT_DIR}
[ $? -eq 0 ] || exit 1
# rm -rf ${BUILD_DIR}/openexr-${OPENEXR_VERSION}
# [ $? -eq 0 ] || exit 1
echo "Done: OpenEXR"
I belive the setup for mingw and mysys is not correct and the system can not find the include (and librry path). How can I check this?
As I noticed from the source code files, openexr should build with cmake tool. You have to do steps of CMake.
Otherwise compile just by g++ or gcc doesn't work.