Cronjob on AWS Elastic Beanstalk Django App - django

Im having issue on running my cron jobs from Django EB.
When i deployed it shows no error but the cronjob is not running.
the import_incontact_statelogs.py is a script that should dump a set of data to my database.
Here's my config file inside .ebextension
-----Cronjob.config--------------
files:
"/usr/local/bin/check_leader_only_instance.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null`
REGION=`curl -s http://169.254.169.254/latest/dynamic/instance-identity/document 2>/dev/null | jq -r .region`
# Find the Auto Scaling Group name from the Elastic Beanstalk environment
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output json | jq -r '.[][] | select(.Key=="aws:autoscaling:groupName") | .Value'`
# Find the first instance in the Auto Scaling Group
FIRST=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output json | \
jq -r '.AutoScalingGroups[].Instances[] | select(.LifecycleState=="InService") | .InstanceId' | sort | head -1`
# If the instance ids are the same exit 0
[ "$FIRST" = "$INSTANCE_ID" ]
"/usr/local/bin/my_cron_script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
/usr/local/bin/check_leader_only_instance.sh || exit
source /opt/python/run/venv/bin/activate
source /opt/python/current/env
cd /opt/python/current/app
python manage.py shell < automations/import_incontact_statelogs.py
"/etc/cron.d/daily_cron":
mode: "000644"
owner: root
group: root
content: |
*/10 * * * * root /usr/local/bin/my_cron_script.sh
commands:
rm_old_cron:
command: "rm -fr /etc/cron.d/*.bak"
ignoreErrors: true

Related

I'm getting "command not found" error after running these commands in GitLab. I used these commands 2 days ago and they worked fine

I was using these commands for my deploy-job the other day and it worked fine. This is a new pipeline for a new project and now these commands aren't working. I'm getting errors in my pipeline after every command saying "command not found". Here's my gitlab-ci file for reference
variables:
DOCKER_REGISTRY: 775362094965.dkr.ecr.us-west-2.amazonaws.com
AWS_DEFAULT_REGION: us-west-2
APP_NAME: flask-app
DOCKER_HOST: tcp://docker:2375
stages:
- build
- deploy
build-job:
stage: build
image:
name: amazon/aws-cli
entrypoint: [""]
services:
- docker:dind
before_script:
- amazon-linux-extras install docker
- aws --version
- docker --version
script:
- docker build -t $DOCKER_REGISTRY/$APP_NAME:latest .
- aws ecr get-login-password | docker login --username AWS --password-stdin $DOCKER_REGISTRY
- docker push $DOCKER_REGISTRY/$APP_NAME:latest
deploy-job:
stage: deploy
script:
- echo `aws ecs describe-task-definition --task-definition $CI_AWS_ECS_TASK_DEFINITION --region us-west-2` > input.json
- echo $(cat input.json | jq '.taskDefinition.containerDefinitions[].image="'$REPOSITORY_URI':'$IMAGE_TAG'"') > input.json
- echo $(cat input.json | jq '.taskDefinition') > input.json
- echo $(cat input.json | jq 'del(.taskDefinitionArn)' | jq 'del(.revision)' | jq 'del(.status)' | jq 'del(.requiresAttributes)' | jq 'del(.compatibilities)' | jq 'del(.registeredAt)' | jq 'del(.registeredBy)') > input.json
- aws ecs register-task-definition --cli-input-json file://input.json --region us-west-2
- revision=$(aws ecs describe-task-definition --task-definition $CI_AWS_ECS_TASK_DEFINITION --region us-west-2 | egrep "revision" | tr "/" " " | awk '{print $2}' | sed 's/"$//' | cut -d "," -f 1)
- aws ecs update-service --cluster $CI_AWS_ECS_CLUSTER --service $CI_AWS_ECS_SERVICE --task-definition $CI_AWS_ECS_TASK_DEFINITION:$revision --region us-west-2
My build-job works fine, I'm just getting "command not found" with my deploy-job.
You need to specify an image outside of the build job or in the deploy job. Right now, you're only specifying an image inside your build-job.

IAM based ssh to EC2 instance using CloudFormation template

I am using an AWS CloudFormation template for IAM role-based access to an EC2 instance.
I getting permission denied error while running the template, and I am not able to access the EC2 machine with a username without a pem file.
Instance:
Type: 'AWS::EC2::Instance'
Metadata:
'AWS::CloudFormation::Init':
config:
files:
/opt/authorized_keys_command.sh:
content: >
#!/bin/bash -e
if [ -z "$1" ]; then
exit 1
fi
SaveUserName="$1"
SaveUserName=${SaveUserName//"+"/".plus."}
SaveUserName=${SaveUserName//"="/".equal."}
SaveUserName=${SaveUserName//","/".comma."}
SaveUserName=${SaveUserName//"#"/".at."}
aws iam list-ssh-public-keys --user-name "$SaveUserName" --query
"SSHPublicKeys[?Status == 'Active'].[SSHPublicKeyId]" --output
text | while read KeyId; do
aws iam get-ssh-public-key --user-name "$SaveUserName" --ssh-public-key-id "$KeyId" --encoding SSH --query "SSHPublicKey.SSHPublicKeyBody" --output text
done
mode: '000755'
owner: root
group: root
/opt/import_users.sh:
content: >
#!/bin/bash
aws iam list-users --query "Users[].[UserName]" --output text |
while read User; do
SaveUserName="$User"
SaveUserName=${SaveUserName//"+"/".plus."}
SaveUserName=${SaveUserName//"="/".equal."}
SaveUserName=${SaveUserName//","/".comma."}
SaveUserName=${SaveUserName//"#"/".at."}
if id -u "$SaveUserName" >/dev/null 2>&1; then
echo "$SaveUserName exists"
else
#sudo will read each file in /etc/sudoers.d, skipping file names that end in ?~? or contain a ?.? character to avoid causing problems with package manager or editor temporary/backup files.
SaveUserFileName=$(echo "$SaveUserName" | tr "." " ")
/usr/sbin/adduser "$SaveUserName"
echo "$SaveUserName ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/$SaveUserFileName"
fi
done
mode: '000755' owner: root group: root
/etc/cron.d/import_users:
content: |
*/10 * * * * root /opt/import_users.sh
mode: '000644' owner: root
group: root
/etc/cfn/cfn-hup.conf:
content: !Sub |
[main]
stack=${AWS::StackId}
region=${AWS::Region}
interval=1
mode: '000400' owner: root group: root
/etc/cfn/hooks.d/cfn-auto-reloader.conf:
content: !Sub >
[cfn-auto-reloader-hook]
triggers=post.update
path=Resources.Instance.Metadata.AWS::CloudFormation::Init
action=/opt/aws/bin/cfn-init --verbose
--stack=${AWS::StackName} --region=${AWS::Region}
--resource=Instance
runas=root
commands:
a_configure_sshd_command:
command: >-
sed -i 's:#AuthorizedKeysCommand none:AuthorizedKeysCommand
/opt/authorized_keys_command.sh:g' /etc/ssh/sshd_config
b_configure_sshd_commanduser:
command: >-
sed -i 's:#AuthorizedKeysCommandUser
nobody:AuthorizedKeysCommandUser nobody:g' /etc/ssh/sshd_config
c_import_users:
command: ./import_users.sh
cwd: /opt
services:
sysvinit:
cfn-hup:
enabled: true
ensureRunning: true
files:
- /etc/cfn/cfn-hup.conf
- /etc/cfn/hooks.d/cfn-auto-reloader.conf
sshd:
enabled: true
ensureRunning: true
commands:
- a_configure_sshd_command
- b_configure_sshd_commanduser
'AWS::CloudFormation::Designer':
id: 85ddeee0-0623-4f50-8872-1872897c812f
Properties:
ImageId: !FindInMap
- RegionMap
- !Ref 'AWS::Region'
- AMI
IamInstanceProfile: !Ref InstanceProfile
InstanceType: t2.micro
UserData:
'Fn::Base64': !Sub >
#!/bin/bash -x
/opt/aws/bin/cfn-init --verbose --stack=${AWS::StackName}
--region=${AWS::Region} --resource=Instance
/opt/aws/bin/cfn-signal --exit-code=$? --stack=${AWS::StackName}
--region=${AWS::Region} --resource=Instance
This User Data script will configure a Linux instance to use password authentication.
While the password here is hard-coded, you could obtain it in other ways and set it to the appropriate value.
#!
echo 'secret-password' | passwd ec2-user --stdin
sed -i 's|[#]*PasswordAuthentication no|PasswordAuthentication yes|g' /etc/ssh/sshd_config
systemctl restart sshd.service

Amazon's AWS ElasticBeanstalk Let's Encrypt CertBot

How can I run and automated scalable version of Let's encrypt Certbot on Elastic beanstalk ? Haven't found an answer on the web so I put together this .ebextension. I hope you find it useful.
Don't use as is, check and modify according to your needs.
Requirements:
ElasticBeanstalk Environmnet
Ec2-elastic-beanstalk-service-role with access to Route53 Resource
TCP Loadbalancer in front of EC2 instances
Deployment policy rolling (one instance at a the time)
NFS mount
It's a single .ebextension:
jq package is needed:
packages:
yum:
jq: []
Creating jsons:
files:
"/usr/local/bin/certbot/add_json.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
#Create TXT record JSON
cat > $1_TXT.json << END
{
"Comment": "TXT Verification for CertBOT",
"Changes": [
{
"Action": "$1",
"ResourceRecordSet": {
"Name": "_acme-challenge.$CERTBOT_DOMAIN",
"Type": "TXT",
"TTL": 300,
"ResourceRecords": [
{
"Value": "\"$CERTBOT_VALIDATION\""
}
]
}
}
]
}
END
Removing hook to Route53:
"/usr/local/bin/certbot/remove_txt_hook.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
PWD=`pwd`
APEX_DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')
ZONE_ID=`aws route53 list-hosted-zones --output text | awk '$4 ~ /^ *'$APEX_DOMAIN'/''{print $3}' | sed 's:.*/::'`
aws route53 change-resource-record-sets \
--hosted-zone-id $ZONE_ID --change-batch file://$PWD/DELETE_TXT.json
rm CREATE_TXT.json
rm DELETE_TXT.json
Adding hook to Route53:
"/usr/local/bin/certbot/add_txt_hook.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
PWD=`pwd`
APEX_DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')
ZONE_ID=`aws route53 list-hosted-zones --output text | awk '$4 ~ /^ *'$APEX_DOMAIN'/''{print $3}' | sed 's:.*/::'`
./add_json.sh CREATE
./add_json.sh DELETE
aws route53 change-resource-record-sets \
--hosted-zone-id $ZONE_ID --change-batch file://$PWD/CREATE_TXT.json
sleep 30
Deploying:
"/usr/local/bin/certbot/start_process.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
MY_DOMAIN=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MY_DOMAIN')
EFS_NAME=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.EFS_NAME')
PWD=`pwd`
if [ "$MY_DOMAIN" = example.com ]; then
if [ ! -d /etc/letsencrypt ]; then
mkdir -p /etc/letsencrypt
fi
if ! grep -qs ' /etc/letsencrypt ' /proc/mounts; then
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_NAME:/.certificates/.letsencrypt /etc/letsencrypt || exit
fi
if [ ! -f certbot-auto ]; then
yum -y install mod24_ssl augeas-libs libffi-devel python27-tools system-rpm-config
wget https://dl.eff.org/certbot-auto
chmod 550 certbot-auto
fi
if [ ! -f /etc/letsencrypt/live/$MY_DOMAIN/fullchain.pem ]; then
./certbot-auto certonly --debug -n --no-bootstrap --email <your e-mail> --agree-tos --manual-public-ip-logging-ok --manual --preferred-challenges=dns --manual-auth-hook $PWD/add_txt_hook.sh --manual-cleanup-hook $PWD/remove_txt_hook.sh -d $MY_DOMAIN
fi
echo "00 15 * * SUN root cd /usr/local/bin/certbot && ./renewal.sh >> certbot.log 2>&1" > /etc/cron.d/cron_certbot
fi
Renewing:
"/usr/local/bin/certbot/renewal.sh":
mode: "000550"
owner: root
group: root
content: |
##!/bin/bash
MY_DOMAIN=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MY_DOMAIN')
PWD=`pwd`
ENV_ID=`{"Ref": "AWSEBEnvironmentId" }`
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
TODAY=`date +%Y-%m-%d`
STATUS=`aws elasticbeanstalk describe-environments --environment-ids $ENV_ID --region $REGION | awk '/"Status"/ {print substr($2, 1, length($2)-1)}' | sed 's/\"//g'`
while [ "$STATUS" != "Ready" ]; do
STATUS=`aws elasticbeanstalk describe-environments --environment-ids $ENV_ID --region $REGION | awk '/"Status"/ {print substr($2, 1, length($2)-1)}' | sed 's/\"//g'`
sleep 10
done
if ! /usr/local/bin/certbot/one_instance.sh; then
i="0"
while [ "$i" -lt 180 ] && [ ! -f /etc/letsencrypt/renewed_$TODAY ]; do
i=$[$i+1]
sleep 2
done
if [ ! -f /etc/letsencrypt/renewed_$TODAY ]; then
exit
else
/etc/init.d/httpd graceful; exit
fi
fi
./certbot-auto renew --debug --no-bootstrap --renew-hook "/etc/init.d/httpd graceful; touch /etc/letsencrypt/renewed_$TODAY; find /etc/letsencrypt/ -type f -name 'renewed_*' -mtime +0 -exec rm {} \;"
Only on one instance:
"/usr/local/bin/certbot/one_instance.sh":
mode: "000550"
owner: root
group: root
content: |
#!/bin/bash
METADATA=/opt/aws/bin/ec2-metadata
INSTANCE_ID=`$METADATA -i | awk '{print $2}'`
REGION=`$METADATA -z | awk '{print substr($2, 0, length($2)-1)}'`
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output text | awk '/aws:autoscaling:groupName/ {print $5}'`
SOLO_I=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output text | awk '/InService/ {print $4}' | sort | head -1`
[ "$SOLO_I" = "$INSTANCE_ID" ]
Executing:
commands:
01_start_certbot_deploy:
command: "/usr/local/bin/certbot/start_process.sh &>> certbot.log"
cwd: "/usr/local/bin/certbot"
02_delete_bak_files:
command: "rm -f *.bak"
cwd: "/usr/local/bin/certbot"
Let me know if you have some remarks or improvement suggestions.
Thanks!

Elastic-Beanstalk: How to find the S3 Bucket From the EB Instance

I'm interested in creating a backup of the current application as an application version. As a result I'd like to try and track down the S3 Bucket where the App Versions are saved to save a new file to that location.
Does anyone know how this is done?
To do this:
Get the Instance Id
Get the environment Id.
Get the App Name
Get the App Versions
The data is in the response of the App Versions.
Based on the above, here is my entire backup script. It will figure out steps 1-4 above, then zip up the current app directory, send it to S3, and create a new app version based on it.
#/bin/bash
EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id || die \"wget instance-id has failed: $?\"`"
test -n "$EC2_INSTANCE_ID" || die 'cannot obtain instance-id'
EC2_AVAIL_ZONE="`wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone || die \"wget availability-zone has failed: $?\"`"
test -n "$EC2_AVAIL_ZONE" || die 'cannot obtain availability-zone'
EC2_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
EB_ENV_ID="`aws ec2 describe-instances --instance-ids $EC2_INSTANCE_ID --region $EC2_REGION | grep -B 1 \"elasticbeanstalk:environment-id\" | grep -Po '\"Value\":\\s+\"[^\"]+\"' | cut -d':' -f 2 | grep -Po '[^\"]+'`"
EB_APP_NAME="`aws elasticbeanstalk describe-environments --environment-id $EB_ENV_ID --region $EC2_REGION | grep 'ApplicationName' | cut -d':' -f 2 | grep -Po '[^\\",]+'`"
EB_BUCKET_NAME="`aws elasticbeanstalk describe-application-versions --application-name $EB_APP_NAME --region $EC2_REGION | grep -m 1 'S3Bucket' | cut -d':' -f 2 | grep -Po '[^\", ]+'`"
DATE=$(date -d "today" +"%Y%m%d%H%M")
FILENAME=application.$DATE.zip
cd /var/app/current
sudo -u webapp zip -r -J /tmp/$FILENAME *
sudo mv /tmp/$FILENAME ~/.
sudo chmod 755 ~/$FILENAME
cd ~
aws s3 cp $FILENAME s3://$EB_BUCKET_NAME/
rm -f ~/$FILENAME
aws elasticbeanstalk create-application-version --application-name $EB_APP_NAME --version-label "$DATE-$FILENAME" --description "Auto-Save $DATE" --source-bundle S3Bucket="$EB_BUCKET_NAME",S3Key="$FILENAME" --no-auto-create-application --region $EC2_REGION
tldr: most probably you want
jq -r '..|.url?' /etc/elasticbeanstalk/metadata-cache | grep -oP '//\K(.*)(?=\.s3)
Detailed version:
Add to any of your .ebextensions
packages:
yum:
jq: []
(assuming you are using Amazon Linux or Centos)
and run
export BUCKET="$(jq -r '..|.url?' /etc/elasticbeanstalk/metadata-cache | grep -oP '//\K(.*)(?=\.s3)' || sudo !!)"
echo bucketname: $BUCKET
then full s3 path would be accessible at
bash -c 'source /etc/elasticbeanstalk/.aws-eb-stack.properties; echo s3:\\${BUCKET} -r ${region:-eu-east-3}'

How do you get access to environment variables via Elasticbeanstalk configuration files (using Docker)?

For example, if I wish to mount a particular volume that is defined by an environment variable.
I ended up using the following code:
---
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/02my_setup.sh":
owner: root
group: root
mode: "000755"
content: |
#!/bin/bash
set -e
. /opt/elasticbeanstalk/hooks/common.sh
EB_CONFIG_APP_CURRENT=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_SUPPORT_FILES_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_files_dir)
# load env vars
eval $($EB_SUPPORT_FILES_DIR/generate_env | sed 's/$/;/')
You can use /opt/elasticbeanstalk/bin/get-config environment in a bash script
Example:
# .ebextensions/efs_mount.config
commands:
01_mount:
command: "/tmp/mount-efs.sh"
files:
"/tmp/mount-efs.sh":
mode: "000755"
content : |
#!/bin/bash
EFS_REGION=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.EFS_REGION')
EFS_MOUNT_DIR=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.EFS_MOUNT_DIR')
EFS_VOLUME_ID=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.EFS_VOLUME_ID')