I am trying to deploy a Sagemaker Lifecycle with AWS CloudFormation.
The Lifecycle is importing ipynb notebooks from s3 bucket to the Sagemaker notebook instance.
the bucket name is specified in the parameters, I want to use it in a !Sub function inside the bash script of the Lifecycle.
The problem is that the CF runs first on the template and tries to complete its own functions (like !Sub) and then the scripts upload as bash script to the Lifecycle.
This is my code:
LifecycleConfig:
Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
Properties:
NotebookInstanceLifecycleConfigName: !Sub
- ${NotebookInstanceName}LifecycleConfig
- NotebookInstanceName: !Ref NotebookInstanceName
OnStart:
- Content:
Fn::Base64: !Sub
- |
#!/bin/bash -xe
set -e
CP_SAMPLES=true
EXTRACT_CSV=false
s3region=s3.amazonaws.com
SRC_NOTEBOOK_DIR=${Consumer2BucketName}/sagemaker-notebooks
Sagedir=/home/ec2-user/SageMaker
industry=industry
notebooks=("notebook1.ipynb" "notebook2.ipynb" "notebook3.ipynb")
download_files(){
for notebook in "${notebooks[#]}"
do
printf "aws s3 cp s3://${SRC_NOTEBOOK_DIR}/${notebook} ${Sagedir}/${industry}\n"
aws s3 cp s3://"${SRC_NOTEBOOK_DIR}"/"${notebook}" ${Sagedir}/${industry}
done
}
if [ ${CP_SAMPLES} = true ]; then
sudo -u ec2-user mkdir -p ${Sagedir}/${industry}
mkdir -p ${Sagedir}/${industry}
download_files
chmod -R 755 ${Sagedir}/${industry}
chown -R ec2-user:ec2-user ${Sagedir}/${industry}/.
fi
- Consumer2BucketName: !Ref Consumer2BucketName
Raised the following error:
Template error: variable names in Fn::Sub syntax must contain only alphanumeric characters, underscores, periods, and colons
It seems that was a conflict with the Bash Vars and the !Sub CF function.
In the following template I changed the Bash Vars and removed the {}:
LifecycleConfig:
Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
Properties:
NotebookInstanceLifecycleConfigName: !Sub
- ${NotebookInstanceName}LifecycleConfig
- NotebookInstanceName: !Ref NotebookInstanceName
OnStart:
- Content:
Fn::Base64:
!Sub
- |
#!/bin/bash -xe
set -e
CP_SAMPLES=true
EXTRACT_CSV=false
s3region=s3.amazonaws.com
SRC_NOTEBOOK_DIR=${Consumer2BucketName}/sagemaker-notebooks
Sagedir=/home/ec2-user/SageMaker
industry=industry
notebooks=("notebook1.ipynb" "notebook2.ipynb" "notebook3.ipynb")
download_files(){
for notebook in $notebooks
do
printf "aws s3 cp s3://$SRC_NOTEBOOK_DIR/${!notebook} $Sagedir/$industry\n"
aws s3 cp s3://"$SRC_NOTEBOOK_DIR"/"${!notebook}" $Sagedir/$industry
done
}
if [ $CP_SAMPLES = true ]; then
sudo -u ec2-user mkdir -p $Sagedir/$industry
mkdir -p $Sagedir/$industry
download_files
chmod -R 755 $Sagedir/$industry
chown -R ec2-user:ec2-user $Sagedir/$industry/.
fi
- Consumer2BucketName: !Ref Consumer2BucketName
The problem here is the for loop is not running through all the notebooks in the list but importing only the first one.
After going through some solutions I tried adding [#] to the notebooks:
for notebook in $notebooks[#]
and
for notebook in “$notebooks[#]“/”$notebooks[*]“/$notebooks[#]
I got the same error.
It seems that was a conflict with the Bash Vars and the !Sub CF function.
That's correct. Both bash and !Sub use ${} for variable substitution. You can escape the bash variables with ${!}. For example:
for notebook in "${!notebooks[#]}"
Also mentioned in the docs:
To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.
Im having issue on running my cron jobs from Django EB.
When i deployed it shows no error but the cronjob is not running.
the import_incontact_statelogs.py is a script that should dump a set of data to my database.
Here's my config file inside .ebextension
-----Cronjob.config--------------
files:
"/usr/local/bin/check_leader_only_instance.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id 2>/dev/null`
REGION=`curl -s http://169.254.169.254/latest/dynamic/instance-identity/document 2>/dev/null | jq -r .region`
# Find the Auto Scaling Group name from the Elastic Beanstalk environment
ASG=`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
--region $REGION --output json | jq -r '.[][] | select(.Key=="aws:autoscaling:groupName") | .Value'`
# Find the first instance in the Auto Scaling Group
FIRST=`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
--region $REGION --output json | \
jq -r '.AutoScalingGroups[].Instances[] | select(.LifecycleState=="InService") | .InstanceId' | sort | head -1`
# If the instance ids are the same exit 0
[ "$FIRST" = "$INSTANCE_ID" ]
"/usr/local/bin/my_cron_script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
/usr/local/bin/check_leader_only_instance.sh || exit
source /opt/python/run/venv/bin/activate
source /opt/python/current/env
cd /opt/python/current/app
python manage.py shell < automations/import_incontact_statelogs.py
"/etc/cron.d/daily_cron":
mode: "000644"
owner: root
group: root
content: |
*/10 * * * * root /usr/local/bin/my_cron_script.sh
commands:
rm_old_cron:
command: "rm -fr /etc/cron.d/*.bak"
ignoreErrors: true
I am trying to create a CloudFormation stack which has UserData script to install java, tomcat, httpd and java application on launch of an EC2 instance.
However, the stack gets created successfully with all the resources but when I connect to EC2 instance to check the configuration of above applications I don't find any. My usecase is to spin-up an instance with all the above applications/software to be installed with automation.
UserData:
Fn::Base64:
Fn::Join:
- ' '
- - '#!/bin/bash -xe\n'
- 'sudo yum update && install pip && pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n'
- 'date > /home/ec2-user/starttime\n'
- 'sudo yum update -y aws-cfn-bootstrap\n'
# Initialize CloudFormation bits\n
- ' '
- '/opt/aws/bin/cfn-init -v\n'
- ' --stack\n'
- '!Ref AWS::StackName\n'
- ' --resource LaunchConfig\n'
- 'ACCESS_KEY=${HostKeys}&SECRET_KEY=${HostKeys.SecretAccessKey}\n'
# Start servers\n
- 'service tomcat8 start\n'
- '/etc/init.d/httpd start\n'
- 'date > /home/ec2-user/stoptime\n'
Metadata:
AWS::CloudFormation::Init:
config:
packages:
yum:
- java-1.8.0-openjdk.x86_64: []
- tomcat8: []
- httpd: []
services:
sysvinit:
httpd:
enabled: 'true'
ensureRunning: 'true'
files:
- /usr/share/tomcat8/webapps/sample.war:
- source: https://s3-eu-west-1.amazonaws.com/testbucket/sample.war
- mode: 000500
- owner: tomcat
- group: tomcat
CfnUser:
Type: AWS::IAM::User
Properties:
Path: '/'
Policies:
- PolicyName: Admin
PolicyDocument:
Statement:
- Effect: Allow
Action: '*'
Resource: '*'
HostKeys:
Type: AWS::IAM::AccessKey
Properties:
UserName: !Ref CfnUser
The problem is in the way you have formatted your UserData. I would suggest that you launch the EC2 instance and manually test the script first. It has a number of problems in it.
Try formatting your UserData like this:
UserData:
Fn::Base64:
!Sub |
#!/bin/bash -xe
# FIXME. This won't work either.
# sudo yum update && install pip && pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
date > /home/ec2-user/starttime
sudo yum update -y aws-cfn-bootstrap
# Initialize CloudFormation bits
/opt/aws/bin/cfn-init -v \
--stack ${AWS::StackName} \
--resource LaunchConfig
# FIXME. Not sure why these are here.
# ACCESS_KEY=${HostKeys}
# SECRET_KEY=${HostKeys.SecretAccessKey}
# Start servers\n
service tomcat8 start
/etc/init.d/httpd start
date > /home/ec2-user/stoptime
Things to note:
You can't interpolate here using !Ref notation. Notice I changed it to ${AWS::StackName} and notice the whole block is inside !Sub.
As my comments indicate, the yum update line has invalid commands in it.
As noted in the comments, it is a bad practice to inject access keys. Also, the keys don't seem to be required for anything in this script.
Note also that the files section is specified incorrectly in the MetaData, as Arrays instead of Hash keys.
It should be:
files:
/usr/share/tomcat8/webapps/sample.war:
source: https://s3-eu-west-1.amazonaws.com/testbucket/sample.war
mode: '000500'
owner: tomcat
group: tomcat
I want to get the content of a file in the UserData without providing inline content as following, but I'm getting the path of the file in the content when the ec2 boots instead of the content of the file.
Here's a snippet of my template:
ServiceInstance:
Type: "AWS::EC2::Instance"
Properties:
. . .
UserData:
'Fn::Base64': !Sub |
#cloud-config
write_files:
- path: /etc/sysconfig/cloudformation
permissions: 0644
owner: root
content: |
STACK_NAME=${AWS::StackName}
AWS_REGION=${AWS::Region}
- path: /etc/path-to-file/conf.yaml
permissions: 0644
owner: root
content: "#file://./config/conf-${Env}.yaml"
runcmd:
## run some commands
when I ssh to ec2 and check the file content I get this:
[ec2-user#ec2ip ~]$ cat /etc/path-to-file/conf.yaml
#file://./config/conf-dev.yaml
I checked this cloud init docs, but can't find something related.
any idea what did I do wrong in here ?
Encode the file content in base64 and pass it as argument. Cloud Init will decode the string b64.
Be careful of the cloudformation size limits on template and variables.
Parameters:
ConfContent:
Type: String
Description: "Conf content in base64 format."
Resources:
ServiceInstance:
Type: "AWS::EC2::Instance"
Properties:
. . .
UserData:
'Fn::Base64': !Sub
- |
#cloud-config
write_files:
- path: /etc/sysconfig/cloudformation
permissions: 0644
owner: root
content: |
STACK_NAME=${AWS::StackName}
AWS_REGION=${AWS::Region}
- path: /etc/path-to-file/conf.yaml
permissions: 0644
owner: root
content: ${CONF_CONTENT}
encoding: b64
runcmd:
## run some commands
- CONF_CONTENT: !Ref ConfContent
Then expose the file content as property :
aws cloudformation create-stack \
--stack-name "mystack" \
--template-body "template.yaml" \
--parameters \
ParameterKey=ConfContent,ParameterValue=\"$(base64 -w0 ./config/conf-dev.yaml)\"
Is there anyway we can pass dynamic references to Secret Manager to AWS Launch Config User Data?
Here is the code snippet I tried:
"SampleLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},
"AMI"
]
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"yum update -y\n",
"useradd -p <<pwd>>{{resolve:secretsmanager:Credentials:SecretString:userName}}\n",
"\n"
]
]
}
}
}
}
Seems error in getting the useradd: invalid user name '{{resolve:secretsmanager:Credentials:SecretString:userName}}'
How can I pass Secret Manager secret value to cloudformation user data ?
It seems that {{resolve:...}} dynamic references are only expanded in certain contexts within a template.
There is no precise information in the AWS docs about exactly where in a template you can use these references. The current wording with regard to {{resolve:secretsmanager:...}} says:
"The secretsmanager dynamic reference can be used in all resource properties"
However this is contradicted by your example, and I've also observed dynamic references failing to resolve inside of CloudFormation::Init data.
I have an active Support case open with AWS about this, they have agreed that the behaviour of dynamic references is inadequately documented. I'll update this answer as I learn more.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
I am not sure why this is not expanded correctly for you. However, you probably do not want CFN to expand your secret in the user data because the password would be embedded in the base64 encoded user data script which is visible in the EC2 console.
Instead you should take advantage of the fact that you have a script that executes on the host and call secrets manager at script execution time (warning untested):
"SampleLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},
"AMI"
]
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"yum update -y\n",
"yum install -y jq\n",
!Sub "useradd -p `aws --region ${AWS::Region} secretsmanager get-secret-value --secret-id Credentials --query SecretString --output text | jq -r .passwordKey` `aws --region ${AWS::Region} secretsmanager get-secret-value --secret-id Credentials --query SecretString --output text | jq -r .userName`\n",
"\n"
]
]
}
}
}
}
This is not ideal since it expands the password on the command line. It might be made more secure by putting the password in a file first and reading it from there and then shredding the file.
I can confirm that #JoeB's "warning untested" answer works, with the caveat that the machine in question must have permission to read the secret. You'll need something like
MyInstancePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: MyPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource: !Join
- ''
- - !Sub "arn:aws:secretsmanager:${AWS::Region}:"
- !Sub "${AWS::AccountId}:secret:Credentials-??????"
Note a couple of things:
Unlike S3 buckets, you can't do arn:aws:secretsmanager:::secret.... If you don't want to explicitly declare the region and account, you need to use a wildcard. Buried at the bottom of Using Identity-based Policies (IAM Policies) for Secrets Manager
If you don't care about the region or account that owns a secret,
you must specify a wildcard character * (not an empty field) for the
region and account ID number fields of the ARN.
Perhaps less important, and less likely to cause unexpected failures, but still worth note:
Using '??????' as a wildcard to match the 6 random characters that
are assigned by Secrets Manager avoids a problem that occurs if you
use the '*' wildcard instead. If you use the syntax
"another_secret_name-*", it matches not just the intended secret with
the 6 random characters, but it also matches
"another_secret_name-a1b2c3". Using the '??????' syntax
enables you to securely grant permissions to a secret that doesn't yet
exist.
Variant on #JoeB's answer:
Resources:
SampleLaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: !FindInMap [ AWSRegionArch2AMI, !Ref: 'AWS::Region', AMI ]
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
exec > >(tee /var/log/user-data.log | logger -t user-data) 2>&1
yum update -y
yum install -y jq
username=$(aws secretsmanager get-secret-value --secret-id Credentials \
--query SecretString \
--region ${AWS::Region} --output text | jq -r .userName)
password=$(aws secretsmanager get-secret-value --secret-id Credentials \
--query SecretString \
--region ${AWS::Region} --output text | jq -r .passwordKey)
useradd -p "$password" $username
UserData in JSON is painful to watch these days.
I've also added a technique to split out the UserData logic to it's own log file, as otherwise it goes in cloud-init.log, which is also painful to read.
The proper way to do it is to call secret manager to get your data, here is how I succeded doing it :
SftpCredsUserPasswordSecret:
Type: 'AWS::SecretsManager::Secret'
Properties:
Name: 'sftp-creds-user-password-secret'
Description: DB Credentials
GenerateSecretString:
SecretStringTemplate: '{"username":"sftpuser"}'
GenerateStringKey: "password"
PasswordLength: 30
ExcludePunctuation: true
Ec2SftpRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument: # Tells that Ec2 can assume this role
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- ec2.amazonaws.com
Policies: # Tells that you can call for the secret value
- PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "secretsmanager:GetSecretValue"
Resource: !Ref SftpCredsUserPasswordSecret
RoleName: 'role-ec2-sftp'
Ec2SftpIamInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: 'instance-profile-ec2-sftp'
Roles:
- !Ref Ec2SftpRole
Ec2Sftp:
Type: AWS::EC2::Instance
Properties:
ImageId: "ami-06ce3edf0cff21f07"
InstanceType: t2.micro
SecurityGroupIds:
- !ImportValue SgBridgeId
- !ImportValue SgSftpId
SubnetId: !ImportValue PublicSubnetAZbId
KeyName: !Ref KeyName
UserData:
Fn::Base64: !Sub
- |
#!/bin/bash
# Get Variables
secret=`aws --region ${AWS::Region} secretsmanager get-secret-value --secret-id ${secretRef} --query SecretString --output text`
sftpuser=`echo "$secret" | sed -n 's/.*"username":["]*\([^(",})]*\)[",}].*/\1/p'`
sftppassword=`echo "$secret" | sed -n 's/.*"password":["]*\([^(",})]*\)[",}].*/\1/p'`
# Create Sftp User
adduser $sftpuser
echo "$sftpuser:$sftppassword" | chpasswd
# Configure sftp connection
echo "" >> /etc/ssh/sshd_config
echo "Match User $sftpuser" >> /etc/ssh/sshd_config
echo " PasswordAuthentication yes" >> /etc/ssh/sshd_config
echo " ForceCommand /usr/libexec/openssh/sftp-server" >> /etc/ssh/sshd_config
# Restart the service
systemctl restart sshd
-
secretRef: !Ref SftpCredsUserPasswordSecret
IamInstanceProfile: !Ref Ec2SftpIamInstanceProfile
Tags:
- Key: Name
Value: 'ec2-sftp'
I have a cfn template similar to below working
Parameters:
ASecret:
Type: String
Default: '{{resolve:secretsmanager:ASecret}}'
-
-
-
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
echo ${ASecret}
AWS CloudFormation enhances the existing dynamic referencing of AWS Systems Manager Parameter Store parameters in CloudFormation templates. You can now reference the latest Systems Manager parameter values in CloudFormation templates without specifying parameter versions.
See more
https://aws.amazon.com/about-aws/whats-new/2021/04/now-reference-latest-aws-systems-manager-parameter-values-in-aws-cloudformation-templates-without-specifying-parameter-versions/