I have a commadelimted list and template snippet like:
AWSTemplateFormatVersion: 2010-09-09
Parameters:
IPWhitelist:
Description: Comma-delimited list of CIDR blocks.
Type: CommaDelimitedList
Default: '1.1.1.1/32, 2.2.2.2/32, 3.3.3.3/32'
Resources:
EC2Instance:
Type: 'AWS::EC2::Instance'
CreationPolicy:
ResourceSignal:
Timeout: PT5M
Metadata:
'AWS::CloudFormation::Init':
configSets:
foobar_setup:
- configure_foo
configure_foo:
commands:
01_config:
command: !Sub |
IFS=', ' read -r -a array <<< "$(echo ${IPWhitelist} | sed -e 's/[][]//g')"
for IP in "${!array[#]}";do echo $IP >> /foo/bar/allowed_ips.txt;done
I'd like to run the following command in the Init type commands key:
IFS=', ' read -r -a array <<< "$(echo ${IPWhitelist} | sed -e 's/[][]//g')"
for IP in "${array[#]}";do echo $IP >> /etc/squid/allowed_ips.txt;done
so for the array as the doc says
To write a dollar sign and curly braces (${}) literally, add an
exclamation point (!) after the open curly brace, such as ${!Literal}.
AWS CloudFormation resolves this text as ${Literal}.
What about the first line? how to substitute cloudformation parameter inside bash command substitution $()?
Error message:
Template contains errors.: Template error: variable IPWhitelist in
Fn::Sub expression does not resolve to a string
Related
I am trying to deploy a Sagemaker Lifecycle with AWS CloudFormation.
The Lifecycle is importing ipynb notebooks from s3 bucket to the Sagemaker notebook instance.
the bucket name is specified in the parameters, I want to use it in a !Sub function inside the bash script of the Lifecycle.
The problem is that the CF runs first on the template and tries to complete its own functions (like !Sub) and then the scripts upload as bash script to the Lifecycle.
This is my code:
LifecycleConfig:
Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
Properties:
NotebookInstanceLifecycleConfigName: !Sub
- ${NotebookInstanceName}LifecycleConfig
- NotebookInstanceName: !Ref NotebookInstanceName
OnStart:
- Content:
Fn::Base64: !Sub
- |
#!/bin/bash -xe
set -e
CP_SAMPLES=true
EXTRACT_CSV=false
s3region=s3.amazonaws.com
SRC_NOTEBOOK_DIR=${Consumer2BucketName}/sagemaker-notebooks
Sagedir=/home/ec2-user/SageMaker
industry=industry
notebooks=("notebook1.ipynb" "notebook2.ipynb" "notebook3.ipynb")
download_files(){
for notebook in "${notebooks[#]}"
do
printf "aws s3 cp s3://${SRC_NOTEBOOK_DIR}/${notebook} ${Sagedir}/${industry}\n"
aws s3 cp s3://"${SRC_NOTEBOOK_DIR}"/"${notebook}" ${Sagedir}/${industry}
done
}
if [ ${CP_SAMPLES} = true ]; then
sudo -u ec2-user mkdir -p ${Sagedir}/${industry}
mkdir -p ${Sagedir}/${industry}
download_files
chmod -R 755 ${Sagedir}/${industry}
chown -R ec2-user:ec2-user ${Sagedir}/${industry}/.
fi
- Consumer2BucketName: !Ref Consumer2BucketName
Raised the following error:
Template error: variable names in Fn::Sub syntax must contain only alphanumeric characters, underscores, periods, and colons
It seems that was a conflict with the Bash Vars and the !Sub CF function.
In the following template I changed the Bash Vars and removed the {}:
LifecycleConfig:
Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
Properties:
NotebookInstanceLifecycleConfigName: !Sub
- ${NotebookInstanceName}LifecycleConfig
- NotebookInstanceName: !Ref NotebookInstanceName
OnStart:
- Content:
Fn::Base64:
!Sub
- |
#!/bin/bash -xe
set -e
CP_SAMPLES=true
EXTRACT_CSV=false
s3region=s3.amazonaws.com
SRC_NOTEBOOK_DIR=${Consumer2BucketName}/sagemaker-notebooks
Sagedir=/home/ec2-user/SageMaker
industry=industry
notebooks=("notebook1.ipynb" "notebook2.ipynb" "notebook3.ipynb")
download_files(){
for notebook in $notebooks
do
printf "aws s3 cp s3://$SRC_NOTEBOOK_DIR/${!notebook} $Sagedir/$industry\n"
aws s3 cp s3://"$SRC_NOTEBOOK_DIR"/"${!notebook}" $Sagedir/$industry
done
}
if [ $CP_SAMPLES = true ]; then
sudo -u ec2-user mkdir -p $Sagedir/$industry
mkdir -p $Sagedir/$industry
download_files
chmod -R 755 $Sagedir/$industry
chown -R ec2-user:ec2-user $Sagedir/$industry/.
fi
- Consumer2BucketName: !Ref Consumer2BucketName
The problem here is the for loop is not running through all the notebooks in the list but importing only the first one.
After going through some solutions I tried adding [#] to the notebooks:
for notebook in $notebooks[#]
and
for notebook in “$notebooks[#]“/”$notebooks[*]“/$notebooks[#]
I got the same error.
It seems that was a conflict with the Bash Vars and the !Sub CF function.
That's correct. Both bash and !Sub use ${} for variable substitution. You can escape the bash variables with ${!}. For example:
for notebook in "${!notebooks[#]}"
Also mentioned in the docs:
To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.
How do I get public IP of EC2 instance and write to a text file in UserData. Tried the following, but as expected it wrote the text literally, rather than resolving it.
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
echo "Public IP: " !GetAtt Bastion.PublicIp > /home/ec2-user/readme.txt
Thanks in advance! (PS : It has to be yaml)
I'm writing the cloudformation template that includes ec2 instance. In userdata block, I want to create a file with some content. In the file, I'm initializing local variable MY_MESSAGE, but next, after the template is deployed this variable is not shown in the file.
original temlate:
EC2Instance:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-03368e982f317ae48
InstanceType: t2.micro
KeyName: ec2
UserData:
!Base64 |
#!/bin/bash
cat <<EOF > /etc/aws-kinesis/start.sh
#!/bin/sh
MY_MESSAGE="Hello World"
echo $MY_MESSAGE
output file in ec2 instance:
#!/bin/sh
MY_MESSAGE="Hello World"
echo
As you can see variable MY_MESSAGE does not exist in echo block.
You can put EOF in quotes: "EOF":
UserData:
!Base64 |
#!/bin/bash
cat <<"EOF" > /etc/aws-kinesis/start.sh
#!/bin/sh
MY_MESSAGE="Hello World"
echo $MY_MESSAGE
EOF
Stuck with the usage of SecureString from AWS Parameter Store. I am trying to refer to the database password as:
DatabasePassword:
Type: AWS::SSM::Parameter::Value<SecureString>
NoEcho: 'true'
Default: /environment/default/database_password
Description: The database admin account password
This throws an error:
An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unrecognized parameter type: SecureString
However, if I refer to this parameter as String instead of SecureString it throws a different error:
An error occurred (ValidationError) when calling the CreateStack operation: Parameters [/environment/default/database_password] referenced by template have types not supported by CloudFormation.
I did try using '{{resolve:ssm-secure:parameter-name:version}}' and it works for database configuration:
MasterUsername: !Ref DatabaseUsername
MasterUserPassword: '{{resolve:ssm-secure:/environment/default/database_password:1}}'
However, I'm using AWS Fargate docker containers where I'm supplying these values as Environment variables:
Environment:
- Name: DATABASE_HOSTNAME
Value: !Ref DatabaseHostname
- Name: DATABASE_USERNAME
Value: !Ref DatabaseUsername
- Name: DATABASE_PASSWORD
Value: '{{resolve:ssm-secure:/environment/default/database_password:1}}'
This throws an error:
An error occurred (ValidationError) when calling the CreateStack operation: SSM Secure reference is not supported in: [AWS::ECS::TaskDefinition/Properties/ContainerDefinitions/Environment]
Unable to use secure strings in my implementation. Is there any workaround to this problem? AWS announced support for SecureString last year, but unable to find the documentation. All I found was to use resolve which only works in some cases.
References:
1
2
CloudFormation does not support SecureString as template parameter type. You can confirm it in the documentation below, let me quote it.
In addition, AWS CloudFormation does not support defining template
parameters as SecureString Systems Manager parameter types. However,
you can specify Secure Strings as parameter values for certain
resources by using dynamic parameter patterns.
Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html#aws-ssm-parameter-types
As you mention you "could" solve it using dynamic parameter patterns, but only a limited amount of resources supports it. ECS and Fargate does not.
Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html
Maybe you can address it using Secrets Manager, instead you set the password as environment variable for you container, your application get the password in runtime from Secrets Manager, this also improves your security, the password will not be in clear text inside the container.
Below you can see one example of this solution, it is not for container, but the "way of work" is the same using environment variable and Secrets Manager.
Reference: https://aws.amazon.com/blogs/security/how-to-securely-provide-database-credentials-to-lambda-functions-by-using-aws-secrets-manager/
The AWS Secrets Manager can be used to obtain secrets for CloudFormation templates, even where they are not things such as database passwords.
Here is a link to the documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
There are 3 parts to a Secrets Manager secret:
The Secret's Name e.g. PROD_DB_PASSWORD
The Secret's Key e.g. DB_PASSWORD
And the actual Secret Value
You would then resolve the above secret in your CloudFormation template using:
'{{resolve:secretsmanager:PROD_DB_PASSWORD:SecretString:DB_PASSWORD}}'
I know this post is quite old, but I came across a situation where I needed to use a SecureString and found both this post and a blog post that describes a workaround. I thought this could help some people.
Original Post Here
Basically, you can create a .conf file in your .ebextensions folder like this:
---
packages:
yum:
bash: []
curl: []
jq: []
perl: []
files:
/opt/elasticbeanstalk/hooks/restartappserver/pre/00_resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/resolve_ssm_environment_variables.sh
/opt/elasticbeanstalk/hooks/appdeploy/pre/00_resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/resolve_ssm_environment_variables.sh
/opt/elasticbeanstalk/hooks/configdeploy/pre/00_resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/resolve_ssm_environment_variables.sh
/usr/local/bin/resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
set -Eeuo pipefail
# Resolve SSM parameter references in the elasticbeanstalk option_settings environment variables.
# SSM parameter references must take the same form used in CloudFormation, see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-ssm-secure-strings
# supported forms are:
# {{resolve:ssm-secure-env:path:version}}
# {{resolve:ssm-secure-env:path}}
# {{resolve:ssm-env:path:version}}
# {{resolve:ssm-env:path}}
# where "path" is the SSM parameter path and "version" is the parameter version.
if [[ -z "${AWS_DEFAULT_REGION:-}" ]]; then
# not set so get from configuration
AWS_DEFAULT_REGION="$(aws configure get region)" || :
fi
if [[ -z "${AWS_DEFAULT_REGION:-}" ]]; then
# not set so get from metadata
AWS_DEFAULT_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)" || :
fi
if [[ -z "${AWS_DEFAULT_REGION:-}" ]]; then
echo "Could not determine region." 1>&2
exit 1
fi
export AWS_DEFAULT_REGION
readonly CONTAINER_CONFIG_FILE="${1:-/opt/elasticbeanstalk/deploy/configuration/containerconfiguration}"
readonly TEMP_CONTAINER_CONFIG_FILE="$(mktemp)"
i=0
for envvar in $(jq -r ".optionsettings[\"aws:elasticbeanstalk:application:environment\"][]" "${CONTAINER_CONFIG_FILE}"); do
envvar="$(echo "${envvar}" | perl -p \
-e 's|{{resolve:ssm(?:-secure)-env:([a-zA-Z0-9_.-/]+?):(\d+?)}}|qx(aws ssm get-parameter-history --name "$1" --with-decryption --query Parameters[?Version==\\\x60$2\\\x60].Value --output text) or die("Failed to get SSM parameter named \"$1\" with version \"$2\"")|eg;' \
-e 's|{{resolve:ssm(?:-secure)-env:([a-zA-Z0-9_.-/]+?)}}|qx(aws ssm get-parameter --name "$1" --with-decryption --query Parameter.Value --output text) or die("Failed to get SSM parameter named \"$1\"")|eg;')"
export envvar
jq ".optionsettings[\"aws:elasticbeanstalk:application:environment\"][${i}]=env.envvar" < "${CONTAINER_CONFIG_FILE}" > "${TEMP_CONTAINER_CONFIG_FILE}"
cp "${TEMP_CONTAINER_CONFIG_FILE}" "${CONTAINER_CONFIG_FILE}"
rm "${TEMP_CONTAINER_CONFIG_FILE}"
((i++)) || :
done
And then you can use it like that in CloudFormation template (or really any way you want, I use it with Terraform). Note that there is an extra -env suffix to distinguish from the native resolver.
---
AWSTemplateFormatVersion: '2010-09-09'
Resoures:
BeanstalkEnvironment:
Type: AWS::ElasticBeanstalk::Environment
Properties:
OptionSettings:
-
Namespace: "aws:elasticbeanstalk:application:environment"
OptionName: SPRING_DATASOURCE_PASSWORD
Value: !Sub "{{resolve:ssm-secure-env:/my/parameter:42}
I have a bash script that I want to run on my instances but I only want the second portion of the script to run if a value is true. Also, I would prefer to not have the if statement take place in the script.
Parameters:
#TestParameter = TRUE
Resource:
UserData:
Fn::Sub: |
echo "This is a test example"
#If TestParameter is true:
echo "Only is parameter is true"
In my opinion, it's better to construct Sub the parameter into the script:
UserData:
Fn::Sub: |
echo "This is a test example"
if ${TestParameter}; then
echo "Only is parameter is true"
fi
But since you don't want to have the if inside the script, you'll have to construct the body of the script in the Cloudformation template. Building strings in CFN is always messy. Try something like this:
UserData:
Fn::Join: ["\n", ["echo 'This is a test example'",
['Fn::If': [!Equals [!Ref TestParameter, "true"], "echo 'Parameter is true'", ""]]]]