Is it possible to fetch instances details like instances id, vpc id, key-pair, sg group etc from cloudformation template from cfn-get-metadata please assist.
If it possible please share ..
cfn-get-metadata only gets info from the metadata section of the cloudformation template. In cfn-init, you can use values from params and maps.
You can use common cloud formation features to refer to the template like !FindInMap or you can use ${}.
Check out this snippet
owner: !FindInMap [ nodes, !Ref nodeType, userName ]
group: !FindInMap [ nodes, !Ref nodeType, userName ]
configure:
commands:
configure_service:
command: /opt/app/configure.sh
test: "test ! -e /etc/systemd/system/${nodeType}.service"
If you need some data from the stack then you could add Outputs for the items you need and run describe stack in your user data.
aws cloudformation describe-stacks --stack-name myteststack
Related
I want to store a secret in AWS secrets manager and retrieve it in a CloudFormation template.
To test it I just put it in the value of a tag -
MainRouteTable:
Properties:
Tags:
- Key: Environment
Value: LIVE
- Key: Name
Value: '{{resolve:secretsmanager:tvs:SecretString:testname}}'
VpcId: !Ref 'VPC'
Type: AWS::EC2::RouteTable
After I run the CloudFormation using the template and the environment is up, the value for the tag "Name" is "{{resolve:secretsmanager:tvs:SecretString:testname}}" and not the actual secret stored in testname.
I have looked all around and can not figure out what is wrong. According to the AWS docs I am doing it properly.
I can retrieve the secret fine from the CLI -
aws secretsmanager --region us-east-1 get-secret-value --secret-id arn:aws:secretsmanager:us-east-1:xxxxxx:secret:tvs-ZVTiDO --query SecretString --output text | jq -r .testname
Any suggestions?
I followed the instructions here - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
SecretString can only be used in few resources and selected properties. Tags are not supported. The supported list is:
AWS::DirectoryService::MicrosoftAD Password
AWS::DirectoryService::SimpleAD Password
AWS::ElastiCache::ReplicationGroup AuthToken
AWS::IAM::User LoginProfile Password
AWS::KinesisFirehose::DeliveryStream
RedshiftDestinationConfiguration Password
AWS::OpsWorks::App Source Password
AWS::OpsWorks::Stack CustomCookbooksSource Password
AWS::OpsWorks::Stack RdsDbInstances DbPassword
AWS::RDS::DBCluster MasterUserPassword
AWS::RDS::DBInstance MasterUserPassword
AWS::Redshift::Cluster MasterUserPassword
as a general rule, secrets will never display in AWS console, e.g. you can't use the im CloudFormation export, tags ect.
Stuck with the usage of SecureString from AWS Parameter Store. I am trying to refer to the database password as:
DatabasePassword:
Type: AWS::SSM::Parameter::Value<SecureString>
NoEcho: 'true'
Default: /environment/default/database_password
Description: The database admin account password
This throws an error:
An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unrecognized parameter type: SecureString
However, if I refer to this parameter as String instead of SecureString it throws a different error:
An error occurred (ValidationError) when calling the CreateStack operation: Parameters [/environment/default/database_password] referenced by template have types not supported by CloudFormation.
I did try using '{{resolve:ssm-secure:parameter-name:version}}' and it works for database configuration:
MasterUsername: !Ref DatabaseUsername
MasterUserPassword: '{{resolve:ssm-secure:/environment/default/database_password:1}}'
However, I'm using AWS Fargate docker containers where I'm supplying these values as Environment variables:
Environment:
- Name: DATABASE_HOSTNAME
Value: !Ref DatabaseHostname
- Name: DATABASE_USERNAME
Value: !Ref DatabaseUsername
- Name: DATABASE_PASSWORD
Value: '{{resolve:ssm-secure:/environment/default/database_password:1}}'
This throws an error:
An error occurred (ValidationError) when calling the CreateStack operation: SSM Secure reference is not supported in: [AWS::ECS::TaskDefinition/Properties/ContainerDefinitions/Environment]
Unable to use secure strings in my implementation. Is there any workaround to this problem? AWS announced support for SecureString last year, but unable to find the documentation. All I found was to use resolve which only works in some cases.
References:
1
2
CloudFormation does not support SecureString as template parameter type. You can confirm it in the documentation below, let me quote it.
In addition, AWS CloudFormation does not support defining template
parameters as SecureString Systems Manager parameter types. However,
you can specify Secure Strings as parameter values for certain
resources by using dynamic parameter patterns.
Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html#aws-ssm-parameter-types
As you mention you "could" solve it using dynamic parameter patterns, but only a limited amount of resources supports it. ECS and Fargate does not.
Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html
Maybe you can address it using Secrets Manager, instead you set the password as environment variable for you container, your application get the password in runtime from Secrets Manager, this also improves your security, the password will not be in clear text inside the container.
Below you can see one example of this solution, it is not for container, but the "way of work" is the same using environment variable and Secrets Manager.
Reference: https://aws.amazon.com/blogs/security/how-to-securely-provide-database-credentials-to-lambda-functions-by-using-aws-secrets-manager/
The AWS Secrets Manager can be used to obtain secrets for CloudFormation templates, even where they are not things such as database passwords.
Here is a link to the documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager
There are 3 parts to a Secrets Manager secret:
The Secret's Name e.g. PROD_DB_PASSWORD
The Secret's Key e.g. DB_PASSWORD
And the actual Secret Value
You would then resolve the above secret in your CloudFormation template using:
'{{resolve:secretsmanager:PROD_DB_PASSWORD:SecretString:DB_PASSWORD}}'
I know this post is quite old, but I came across a situation where I needed to use a SecureString and found both this post and a blog post that describes a workaround. I thought this could help some people.
Original Post Here
Basically, you can create a .conf file in your .ebextensions folder like this:
---
packages:
yum:
bash: []
curl: []
jq: []
perl: []
files:
/opt/elasticbeanstalk/hooks/restartappserver/pre/00_resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/resolve_ssm_environment_variables.sh
/opt/elasticbeanstalk/hooks/appdeploy/pre/00_resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/resolve_ssm_environment_variables.sh
/opt/elasticbeanstalk/hooks/configdeploy/pre/00_resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
/usr/local/bin/resolve_ssm_environment_variables.sh
/usr/local/bin/resolve_ssm_environment_variables.sh:
mode: "000700"
owner: root
group: root
content: |
#!/usr/bin/env bash
set -Eeuo pipefail
# Resolve SSM parameter references in the elasticbeanstalk option_settings environment variables.
# SSM parameter references must take the same form used in CloudFormation, see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-ssm-secure-strings
# supported forms are:
# {{resolve:ssm-secure-env:path:version}}
# {{resolve:ssm-secure-env:path}}
# {{resolve:ssm-env:path:version}}
# {{resolve:ssm-env:path}}
# where "path" is the SSM parameter path and "version" is the parameter version.
if [[ -z "${AWS_DEFAULT_REGION:-}" ]]; then
# not set so get from configuration
AWS_DEFAULT_REGION="$(aws configure get region)" || :
fi
if [[ -z "${AWS_DEFAULT_REGION:-}" ]]; then
# not set so get from metadata
AWS_DEFAULT_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)" || :
fi
if [[ -z "${AWS_DEFAULT_REGION:-}" ]]; then
echo "Could not determine region." 1>&2
exit 1
fi
export AWS_DEFAULT_REGION
readonly CONTAINER_CONFIG_FILE="${1:-/opt/elasticbeanstalk/deploy/configuration/containerconfiguration}"
readonly TEMP_CONTAINER_CONFIG_FILE="$(mktemp)"
i=0
for envvar in $(jq -r ".optionsettings[\"aws:elasticbeanstalk:application:environment\"][]" "${CONTAINER_CONFIG_FILE}"); do
envvar="$(echo "${envvar}" | perl -p \
-e 's|{{resolve:ssm(?:-secure)-env:([a-zA-Z0-9_.-/]+?):(\d+?)}}|qx(aws ssm get-parameter-history --name "$1" --with-decryption --query Parameters[?Version==\\\x60$2\\\x60].Value --output text) or die("Failed to get SSM parameter named \"$1\" with version \"$2\"")|eg;' \
-e 's|{{resolve:ssm(?:-secure)-env:([a-zA-Z0-9_.-/]+?)}}|qx(aws ssm get-parameter --name "$1" --with-decryption --query Parameter.Value --output text) or die("Failed to get SSM parameter named \"$1\"")|eg;')"
export envvar
jq ".optionsettings[\"aws:elasticbeanstalk:application:environment\"][${i}]=env.envvar" < "${CONTAINER_CONFIG_FILE}" > "${TEMP_CONTAINER_CONFIG_FILE}"
cp "${TEMP_CONTAINER_CONFIG_FILE}" "${CONTAINER_CONFIG_FILE}"
rm "${TEMP_CONTAINER_CONFIG_FILE}"
((i++)) || :
done
And then you can use it like that in CloudFormation template (or really any way you want, I use it with Terraform). Note that there is an extra -env suffix to distinguish from the native resolver.
---
AWSTemplateFormatVersion: '2010-09-09'
Resoures:
BeanstalkEnvironment:
Type: AWS::ElasticBeanstalk::Environment
Properties:
OptionSettings:
-
Namespace: "aws:elasticbeanstalk:application:environment"
OptionName: SPRING_DATASOURCE_PASSWORD
Value: !Sub "{{resolve:ssm-secure-env:/my/parameter:42}
I have to update the content in resolv.conf based upon the region selection through cloudformation template.
Example nameserver 0.0.0.0
the above content need to add in resolv.conf based upon region If I will select US then the nameserver will be different and if it is EU then it will be different
I need how to write this condition in cloudformation template.
Please help me in this.
Thanks
My suggestion would be to put it in the userdata and put a mapping to map the appropriate region to nameserver. Something like
UserData:
Fn::Base64: !Sub
- |
#!/bin/bash
nameserver=${nameserver}
echo "nameserver $nameserver" >> /etc/resolvconf/resolv.conf.d/base
- nameserver: !FindInMap
- RegionMap
- !Ref "AWS::Region"
- nameserver
You will need to declare a RegionMap in mappings to map each region with the appropriate nameserver.
I want to read the URL of my database from parameter store in my CloudFormation template. This is easy enough for a single URL, but I can't figure out how to change the URL with different environments.
I have four environments (development, integration, pre-production and production) and their details are stored on four different paths in Parameter Store:
/database/dev/url
/database/int/url
/database/ppe/url
/database/prod/url
I now want to pick the correct Database URL when deploying via CloudFormation. How can I do this?
Parameters:
Environment:
Type: String
Default: dev
AllowedValues:
- dev
- int
- ppe
- prod
DatabaseUrl:
Type: 'AWS::SSM::Parameter::Value<String>'
# Obviously the '+' operator here won't work - so what do I do?
Default: '/database/' + Environment + '/url'
This feature isn't as neat as one would wish. You have to actually pass name/path of each parameter that you want to look up from the parameter store.
Template:
AWSTemplateFormatVersion: 2010-09-09
Description: Example
Parameters:
BucketNameSuffix:
Type: AWS::SSM::Parameter::Value<String>
Default: /example/dev/BucketNameSuffix
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub parameter-store-example-${BucketNameSuffix}
If you don't pass any parameters to the template, BucketNameSuffix will be populated with the value stored at /example/dev/BucketNameSuffix. If you want to use, say, a prod value (pointed to by /example/prod/BucketNameSuffix), then you should specify value for parameter BucketNameSuffix, but instead of passing the actual value you should pass the alternative name of the parameter to use, so you'd pass /example/prod/BucketNameSuffix.
aws cloudformation update-stack --stack-name example-dev \
--template-body file://./example-stack.yml
aws cloudformation update-stack --stack-name example-prod \
--template-body file://./example-stack.yml \
--parameters ParameterKey=BucketNameSuffix,ParameterValue=/example/prod/BucketNameSuffix
A not so great AWS blog post on this: https://aws.amazon.com/blogs/mt/integrating-aws-cloudformation-with-aws-systems-manager-parameter-store/
Because passing million meaningless parameters seems stupid, I might actually generate an environment-specific template and set the right Default: in the generated template so for prod environment Default would be /example/prod/BucketNameSuffix and then I can update the prod stack without passing any parameters.
You can populate CloudFormation templates with parameters stored in AWS Systems Manager Parameter Store using Dynamic References
In this contrived example we make two lookups using resolve:ssm and replace the environment using !Join and !Sub
AWSTemplateFormatVersion: 2010-09-09
Parameters:
Environment:
Type: String
Default: prod
AllowedValues:
- prod
- staging
Resources:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
Image: !Join
- ''
- - 'docker.io/bitnami/redis#'
- !Sub '{{resolve:ssm:/app/${Environment}/digest-sha/redis}}'
Environment:
- Name: DB_HOST
Value: !Sub '{{resolve:ssm:/app/${Environment}/db-host}}'
You can make use of Fn::Join here.
Here is some pseudo code.
You will have to take Environment as an parameter, which you are already doing.
Creating the required string in the resources where DatabaseUrl is required.
Resources :
Instance :
Type : 'AWS::Some::Resource'
Properties :
DatabaseURL : !Join [ "", [ "/database/", !Ref "Environment" , "/url ] ]
Hope this helps.
Note: You cannot assign value to a Parameter dynamically using some computation logic. All the values for defined parameters should be given as Input.
I like Fn::Sub, which is much cleaner and easy to read.
!Sub "/database/${Environment}/url"
I am stuck into the same problem, below are my findings:
We can compose values and descriptions while writing into the SSM Parameter Store from CloudFormation like below :
LambdaARN:
Type: AWS::SSM::Parameter
Properties:
Type: String
Description: !Sub "Lambda ARN from ${AWS::StackName}"
Name: !Sub "/secure/${InstallationId}/${AWS::StackName}/lambda-function-arn"
Value: !GetAtt LambdaFunction.Arn
We can not compose values/defaults to looking for in SSM Parameter Store. Like below:
Parameters:
...
LambdaARN:
Type: Type: AWS::SSM::Parameter::Value<String>
Value: !Sub "/secure/${InstallationId}/teststack/lambda-function-arn"
This is not allowed as per AWS Documentation[1]. Both(Sub/Join) Functions won't work. Following is the error which I was getting:
An error occurred (ValidationError) when calling the CreateChangeSet
operation: Template format error: Every Default member must be a
string.
Composing and passing values while creating stacks can be done like this:
Parameters:
...
LambdaARN:
Type: Type: AWS::SSM::Parameter::Value<String>
....
$ aws cloudformation deploy --template-file cfn.yml --stack-name mystack --parameter-overrides 'LambdaARN=/secure/devtest/teststack/lambda_function-arn'
If you add your custom tags while putting the values in the Parameter Store, it will overwrite the default tags added by CFN.
Default Tags:
- aws:cloudformation:stack-id
- aws:cloudformation:stack-name
- aws:cloudformation:logical-id
Every time we update the values in parameter store, it creates a new version, which is beneficial when we are using DynamicResolvers, this can serve as a solution to the problem in the question like
{{ resolve:ssm:/my/value:1}}
The last field is the version. Where different versions can point to different environments.
We are using versions with the parameters, and adding the labels to them[2], this can't be done via CFN[3], only possible way via CLI or AWS Console. This is AWS's way of handling multiple environments.
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html
[2] https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-labels.html
[3] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ssm-parameter.html
I'm trying to pass in a list of Subnet IDs to a Cloud formation template, but I get an error. What have I done wrong?
Parameter in template:
ClusterSubnets:
Description: Subnets, in the same VPC where cluster ec2 instances reside.
Typically private. Use mutiples, each in a different AZ for HA.
Type: "List<AWS::EC2::Subnet::Id>"
Parameter snippet from json file:
{
"ParameterKey": "ClusterSubnets",
"ParameterValue": [ "subnet-8fc8c4f7" ]
},
Results In:
Parameter validation failed:
Invalid type for parameter Parameters[1].ParameterValue, value: [u'subnet-8fc8c4f7'], type: type 'list', valid types: type 'basestring'
I know this thread is a year old, an I am assuming you were using the aws cli to create the stack, but I was having the same issue and finally stumbled on the answer. Giving credit where its due:
https://github.com/aws/aws-cli/issues/2478#issuecomment-427167512
escape the "," with a backslash and it will solve the issue. In my case this was in a jekins job bashing out so I had to double backslash:
sh "aws cloudformation create-stack --template-url ${TEMPLATE} \
--stack-name $NAME \
--capabilities CAPABILITY_IAM \
--parameters ParameterKey=SecurityGroups,ParameterValue=\"$SG_GROUP2\\,$SG_GROUP1\"; \
aws cloudformation wait stack-create-complete --stack-name $CLUSTER_NAME"
You should specify the ParameterValue using a comma-delimited list of the subnet IDs you want to include, without spaces. See the section under "list" here:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html