I have a set of strings in Cloudformation, this is how they appear on the command line:
arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app0/yyy
arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app4/yyy
arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app1/yyy
arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app2/yyy
arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app3/yyy
I need to split on '\n' (the new line character)
The strings were generated by CF's CommandRunner:
GetElbv2Arns:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Role: InfrastructureManagement
LogGroup: !Ref LogGroup
SubnetId: !Ref subnetId1
Command:
!Sub
- |
aws elbv2 describe-load-balancers \
--region ${AWS::Region} \
--query "LoadBalancers[].[LoadBalancerArn]" \
--output text \
> /command-output.txt
- dummy: ""
Turns out that event though the value started with '\n' as new line, it is actually just '\n' as text by the time CF gets the value. This is what is displayed in CF's Output pane:
arn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app0/yyy\narn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app4/yyy\narn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app1/yyy\narn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app2/yyy\narn:aws:elasticloadbalancing:us-east-1:xxx:loadbalancer/app/app3/yyy
The way to parse the values is to escape the '\':
Outputs:
ARNs1:
Description: "elbv2 Arns-0"
Value:
!Select [0, !Split [ "\\n", !GetAtt GetElbv2Arns.Output] ]
Related
I want to store a secret in AWS secrets manager and retrieve it in a CloudFormation template.
To test it I just put it in the value of a tag -
MainRouteTable:
Properties:
Tags:
- Key: Environment
Value: LIVE
- Key: Name
Value: '{{resolve:secretsmanager:tvs:SecretString:testname}}'
VpcId: !Ref 'VPC'
Type: AWS::EC2::RouteTable
After I run the CloudFormation using the template and the environment is up, the value for the tag "Name" is "{{resolve:secretsmanager:tvs:SecretString:testname}}" and not the actual secret stored in testname.
I have looked all around and can not figure out what is wrong. According to the AWS docs I am doing it properly.
I can retrieve the secret fine from the CLI -
aws secretsmanager --region us-east-1 get-secret-value --secret-id arn:aws:secretsmanager:us-east-1:xxxxxx:secret:tvs-ZVTiDO --query SecretString --output text | jq -r .testname
Any suggestions?
I followed the instructions here - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
SecretString can only be used in few resources and selected properties. Tags are not supported. The supported list is:
AWS::DirectoryService::MicrosoftAD Password
AWS::DirectoryService::SimpleAD Password
AWS::ElastiCache::ReplicationGroup AuthToken
AWS::IAM::User LoginProfile Password
AWS::KinesisFirehose::DeliveryStream
RedshiftDestinationConfiguration Password
AWS::OpsWorks::App Source Password
AWS::OpsWorks::Stack CustomCookbooksSource Password
AWS::OpsWorks::Stack RdsDbInstances DbPassword
AWS::RDS::DBCluster MasterUserPassword
AWS::RDS::DBInstance MasterUserPassword
AWS::Redshift::Cluster MasterUserPassword
as a general rule, secrets will never display in AWS console, e.g. you can't use the im CloudFormation export, tags ect.
I am getting this error when trying to use !Sub with variables in UserData in CloudFormation:
Template error: One or more Fn::Sub intrinsic functions don't specify expected arguments. Specify a string as first argument, and an optional second argument to specify a mapping of values to replace in the string
Why do I get this error?
This is my code:
LinuxEC2Instance:
Type: AWS::EC2::Instance
Properties:
UserData:
Fn::Base64: !Sub
- arn_id: !If [TestEnvironment, 'id1', 'id2']
- key: !If [TestEnvironment, 'key1', 'key2']
- |
ARN_ID=${arn_id}
KEY=${key}
echo $ARN_ID
echo $KEY
The first argument to Sub must be string. Thus you should change order in your UserData. For example:
Fn::Base64:
!Sub
- |
#!/bin/bash -xe
ARN_ID=${arn_id}
KEY=${key}
echo $ARN_ID
echo $KEY
- arn_id: !If [TestEnvironment, 'id1', 'id2']
key: !If [TestEnvironment, 'key1', 'key2']
Is there anyway we can pass dynamic references to Secret Manager to AWS Launch Config User Data?
Here is the code snippet I tried:
"SampleLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},
"AMI"
]
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"yum update -y\n",
"useradd -p <<pwd>>{{resolve:secretsmanager:Credentials:SecretString:userName}}\n",
"\n"
]
]
}
}
}
}
Seems error in getting the useradd: invalid user name '{{resolve:secretsmanager:Credentials:SecretString:userName}}'
How can I pass Secret Manager secret value to cloudformation user data ?
It seems that {{resolve:...}} dynamic references are only expanded in certain contexts within a template.
There is no precise information in the AWS docs about exactly where in a template you can use these references. The current wording with regard to {{resolve:secretsmanager:...}} says:
"The secretsmanager dynamic reference can be used in all resource properties"
However this is contradicted by your example, and I've also observed dynamic references failing to resolve inside of CloudFormation::Init data.
I have an active Support case open with AWS about this, they have agreed that the behaviour of dynamic references is inadequately documented. I'll update this answer as I learn more.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
I am not sure why this is not expanded correctly for you. However, you probably do not want CFN to expand your secret in the user data because the password would be embedded in the base64 encoded user data script which is visible in the EC2 console.
Instead you should take advantage of the fact that you have a script that executes on the host and call secrets manager at script execution time (warning untested):
"SampleLaunchConfig": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},
"AMI"
]
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"yum update -y\n",
"yum install -y jq\n",
!Sub "useradd -p `aws --region ${AWS::Region} secretsmanager get-secret-value --secret-id Credentials --query SecretString --output text | jq -r .passwordKey` `aws --region ${AWS::Region} secretsmanager get-secret-value --secret-id Credentials --query SecretString --output text | jq -r .userName`\n",
"\n"
]
]
}
}
}
}
This is not ideal since it expands the password on the command line. It might be made more secure by putting the password in a file first and reading it from there and then shredding the file.
I can confirm that #JoeB's "warning untested" answer works, with the caveat that the machine in question must have permission to read the secret. You'll need something like
MyInstancePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: MyPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource: !Join
- ''
- - !Sub "arn:aws:secretsmanager:${AWS::Region}:"
- !Sub "${AWS::AccountId}:secret:Credentials-??????"
Note a couple of things:
Unlike S3 buckets, you can't do arn:aws:secretsmanager:::secret.... If you don't want to explicitly declare the region and account, you need to use a wildcard. Buried at the bottom of Using Identity-based Policies (IAM Policies) for Secrets Manager
If you don't care about the region or account that owns a secret,
you must specify a wildcard character * (not an empty field) for the
region and account ID number fields of the ARN.
Perhaps less important, and less likely to cause unexpected failures, but still worth note:
Using '??????' as a wildcard to match the 6 random characters that
are assigned by Secrets Manager avoids a problem that occurs if you
use the '*' wildcard instead. If you use the syntax
"another_secret_name-*", it matches not just the intended secret with
the 6 random characters, but it also matches
"another_secret_name-a1b2c3". Using the '??????' syntax
enables you to securely grant permissions to a secret that doesn't yet
exist.
Variant on #JoeB's answer:
Resources:
SampleLaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: !FindInMap [ AWSRegionArch2AMI, !Ref: 'AWS::Region', AMI ]
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
exec > >(tee /var/log/user-data.log | logger -t user-data) 2>&1
yum update -y
yum install -y jq
username=$(aws secretsmanager get-secret-value --secret-id Credentials \
--query SecretString \
--region ${AWS::Region} --output text | jq -r .userName)
password=$(aws secretsmanager get-secret-value --secret-id Credentials \
--query SecretString \
--region ${AWS::Region} --output text | jq -r .passwordKey)
useradd -p "$password" $username
UserData in JSON is painful to watch these days.
I've also added a technique to split out the UserData logic to it's own log file, as otherwise it goes in cloud-init.log, which is also painful to read.
The proper way to do it is to call secret manager to get your data, here is how I succeded doing it :
SftpCredsUserPasswordSecret:
Type: 'AWS::SecretsManager::Secret'
Properties:
Name: 'sftp-creds-user-password-secret'
Description: DB Credentials
GenerateSecretString:
SecretStringTemplate: '{"username":"sftpuser"}'
GenerateStringKey: "password"
PasswordLength: 30
ExcludePunctuation: true
Ec2SftpRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument: # Tells that Ec2 can assume this role
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- ec2.amazonaws.com
Policies: # Tells that you can call for the secret value
- PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "secretsmanager:GetSecretValue"
Resource: !Ref SftpCredsUserPasswordSecret
RoleName: 'role-ec2-sftp'
Ec2SftpIamInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: 'instance-profile-ec2-sftp'
Roles:
- !Ref Ec2SftpRole
Ec2Sftp:
Type: AWS::EC2::Instance
Properties:
ImageId: "ami-06ce3edf0cff21f07"
InstanceType: t2.micro
SecurityGroupIds:
- !ImportValue SgBridgeId
- !ImportValue SgSftpId
SubnetId: !ImportValue PublicSubnetAZbId
KeyName: !Ref KeyName
UserData:
Fn::Base64: !Sub
- |
#!/bin/bash
# Get Variables
secret=`aws --region ${AWS::Region} secretsmanager get-secret-value --secret-id ${secretRef} --query SecretString --output text`
sftpuser=`echo "$secret" | sed -n 's/.*"username":["]*\([^(",})]*\)[",}].*/\1/p'`
sftppassword=`echo "$secret" | sed -n 's/.*"password":["]*\([^(",})]*\)[",}].*/\1/p'`
# Create Sftp User
adduser $sftpuser
echo "$sftpuser:$sftppassword" | chpasswd
# Configure sftp connection
echo "" >> /etc/ssh/sshd_config
echo "Match User $sftpuser" >> /etc/ssh/sshd_config
echo " PasswordAuthentication yes" >> /etc/ssh/sshd_config
echo " ForceCommand /usr/libexec/openssh/sftp-server" >> /etc/ssh/sshd_config
# Restart the service
systemctl restart sshd
-
secretRef: !Ref SftpCredsUserPasswordSecret
IamInstanceProfile: !Ref Ec2SftpIamInstanceProfile
Tags:
- Key: Name
Value: 'ec2-sftp'
I have a cfn template similar to below working
Parameters:
ASecret:
Type: String
Default: '{{resolve:secretsmanager:ASecret}}'
-
-
-
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
echo ${ASecret}
AWS CloudFormation enhances the existing dynamic referencing of AWS Systems Manager Parameter Store parameters in CloudFormation templates. You can now reference the latest Systems Manager parameter values in CloudFormation templates without specifying parameter versions.
See more
https://aws.amazon.com/about-aws/whats-new/2021/04/now-reference-latest-aws-systems-manager-parameter-values-in-aws-cloudformation-templates-without-specifying-parameter-versions/
I created two Amazon EC2 instances in AWS CloudFormation using a YAML template. I want to take private IP address of one EC2 instance to the other EC2 instance which has a public IP address. As per AWS documentation we can do that using !GetAtt JMeterServer1Instance.PrivateIp
I want to know under which section of the public EC2 instance I should add that in the template. (Please consider this is a YAML template.)
How do I check that we have received it?
It appears that your requirement is:
Create two instances in a CloudFormation template
In the User Data for Instance-A, refer to Instance-B
This is quite simple. First, define that Instance-B DependsOn Instance-A to ensure the creation of Instance-A before Instance-B.
Then, in the User Data for Instance-B, refer to Instance-A:
UserData:
"Fn::Base64":
!Sub |
#!/bin/bash
echo "${InstanceA.PrivateIp}" >foo
A 'better' method would be to use DNS names with a Hosted Zone for VPC in Route 53. This would create a DNS zone for the VPC, then define a DNS name that can be resolved locally. Link it to Instance-B and then Instance-A could refer to Instance-B by DNS name rather than IP address. This allows the DNS name to point to a different instance in future if desired, and creates less dependencies between Instance-A and Instance-B. (But, admittedly, more setup.)
As per the AWS document
Fn::GetAtt
will do the trick here.
My case is:
EC2Instance001 needs to be created 1st
EC2Instance002 needs to use IP of EC2Instance001.
At EC2Instance002 instance is created with two specific settings:
"DependsOn": [ "EC2Instance001"] as I want EC2Instance001 to be created first.
Under Userdata (or metadata) use { "Fn::GetAtt" : [ "EC2Instance001", "PrivateIp" ] } for getting IP of 1st Instance (EC2Instance001)
Here is how I achieved it (EC2Instance002):
---
EC2Instance002:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Init:
configSets:
InstallAndRun:
- Install
- Configure
Install:
packages:
yum:
git: []
files:
"/tmp/bootstrap.sh":
content:
Fn::Join:
- ''
- - "#!/bin/bash\n"
- 'set -x
'
- 'echo "============================"
'
- 'sudo hostname >> /tmp/EC2Instance.txt
'
- MASTERIP=
- Fn::GetAtt:
- EC2Instance001
- PrivateIp
- "\n"
- "echo $MASTERIP > masterIP.txt \n"
mode: '755'
owner: ec2-user
group: ec2-user
Configure:
commands:
runBootstrapScript:
command: "./bootstrap.sh"
cwd: "/tmp"
DependsOn:
- EC2Instance001
Properties:
InstanceType:
Ref: InstanceType
SecurityGroups:
- Ref: InstanceSecurityGroup
KeyName:
Ref: KeyName
UserData:
Fn::Base64:
Fn::Join:
- ''
- - "#!/bin/bash -xe\n"
- 'yum install -y aws-cfn-bootstrap
'
- "# Install the files and packages from the metadata\n"
- "/opt/aws/bin/cfn-init -v"
- " --stack "
- Ref: AWS::StackName
- " --resource EC2Instance002 "
- " --configsets InstallAndRun "
- " --region "
- Ref: AWS::Region
- "\n"
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
You can see that under Metadata I am capturing IP of EC2Instance001 instance in variable $MASTERIP.
NOTE: Same line in JSON will be written as:
"MASTERIP=",{ "Fn::GetAtt" : [ "EC2Instance001", "PrivateIp" ] }, "\n",
It depends on what you'd like to with the private IP on the other machine.
If you'd like to use it in a script on the other VM, pass it down in the user data script like in this example: UserData script with Resource Attribute CloudFormation
The example on the link is showing the attribute of a NetworkInterface instead of an instance attribute, but it's the same with !GetAtt JMeterServer1Instance.PrivateIp
Ok so reading the documentation on the The intrinsic function Fn::Sub. I can use a literal block to specify the user data script.
UserData:
"Fn::Base64":
!Sub |
#!/bin/bash -xe
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfig --configsets wordpress_install --region ${AWS::Region}
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource WebServerGroup --region ${AWS::Region}
and I can uses a mapping to substitute the ${Domain} variable with the resulting value from the Ref function.
Name: !Sub
- www.${Domain}
- { Domain: !Ref RootDomainName }
But what if I need to use a mapping substitute inside a literal block? Like for example:
"Fn::Base64": !Sub |
<powershell>
Write-host "My Domain is www.${Domain},{ Domain: !Ref RootDomainName }"
</powershell>
This example does not work, and I haven't been unable to find a method that does. Any ideas? The first example makes userdata scripts much easier to write and looks cleaner, but without being able to to use !Ref or !Findinmap it's usefulness is reduced dramatically.
Anyone got any ideas?
Since I arrived to this page through Google, and then found the solution through a different wording here(literally):
How to use !FindInMap in !Sub | userdata section
I'll just add it to save some frustrated searching to others.
Essentially, you have to write your example using the 2nd syntax but a bit more verbosely:
Fn::Base64:
Fn::Sub:
- |+
<powershell>
Write-host "My Domain is www.${Domain}"
</powershell>
- Domain:
Fn::Ref: RootDomainName
You may be able to shorten it a bit, but as the original poster said, mind your commas, quoting and usage of short forms.
P.S.: If the first solution has already served your purpose, you should mark it thus.
In that case you would write it simply as ${RootDomainName}. Local resouces in the same stack can just be mapped by using their resource name.