I got AWS sample serverless app in which I have single yaml files along with multiples handlers.
The problem is that template.yaml is keep growing, how can separate yaml for each handler or group of handlers so it would be easy to manage.
For our projects we started dividing the main YAML into multiple on in a next way:
all lambdas are still described in serverless.yml file
in serverless.yml
resources:
- ${file(./sls-config/cognito-user-pools-authorizer.yml)}
- ${file(./sls-config/aurora.yml)}
- ${file(./sls-config/bucket.yml)}
- ${file(./sls-config/queues.yml)}
- ${file(./sls-config/alarms.yml)}
- ${file(./sls-config/roles.yml)}
- ${file(./sls-config/outputs.yml)}
Yep, splitting of the resources up to you.
alarm.yml
Resources:
SQSAlarmTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: ${self:provider.prefix}-sqs-alarm-topic
TopicName: ${self:provider.prefix}-sqs-alarm-topic
Subscription:
- Endpoint: example-email#mail.com
Protocol: email
Tags: ${self:custom.sqsTags}
and so on.
outputs.yml
Outputs:
CognitoUserPoolId:
Value: ${self:custom.userPool}
CognitoUserPoolClientId:
Value: ${self:custom.userPoolClientId}
DSClusterID:
Description: "RDS Cluster "
Value: { Ref: RDSCluster }
DBAddress:
Value: !GetAtt RDSCluster.Endpoint.Address
The variables from custom and provide may be used easily through the sub_configs.yml.
Be careful with spacing/paddings in yaml files :)
Related
I'm trying to create a AWS::MSK::Configuration resource, as described here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html.
This is my CF template:
Resources:
MSKConfig:
Type: AWS::MSK::Configuration
Properties:
Description: Basic configuration.
Name: test-msk-configuration
ServerProperties: |
auto.create.topics.enable=false
........
It doesn't work failing with the following error:
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: #Unrecognized resource types: [AWS::MSK::Configuration]
I'm not able to find any example with this resource type online. Has anyone ever used it?
Update on 26/03/2022:
I was able to get this deployed. I use the serverless framework for deployment.
MSK-Cluster.yml:
Resources:
ServerlessMSK:
Type: AWS::MSK::Cluster
Properties:
ClusterName: ${self:service}-${self:provider.stage}-msk
KafkaVersion: 2.6.2
BrokerNodeGroupInfo:
InstanceType: kafka.t3.small
ClientSubnets:
- !Ref ServerlessPrivateSubnet1
- !Ref ServerlessPrivateSubnet2
- !Ref ServerlessPrivateSubnet3
SecurityGroups:
- !GetAtt ServerlessMSKSecurityGroup.GroupId
StorageInfo:
EBSStorageInfo:
VolumeSize: 10
NumberOfBrokerNodes: 3
EncryptionInfo:
EncryptionInTransit:
ClientBroker: TLS
InCluster: true
EnhancedMonitoring: PER_TOPIC_PER_BROKER
ConfigurationInfo:
Arn: !GetAtt ServerlessMSKConfiguration.Arn
Revision: 1
MSK-config.yml
Resources:
ServerlessMSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
Description: cluster for msk cluster-${sls:stage}
Name: node-mongo-kafka-experiment-${sls:stage}-config
ServerProperties: ${file('./assets/server.properties')}
server.properties
auto.create.topics.enable=true
default.replication.factor=2
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=10
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000
basically, no base64 was used. I just referenced the file in the deployment and managed to get it up. Hope this helps.
Orginal Answer Below:
I haven't been able to deploy this correctly too. But maybe I can point you in the right direction.I always get back a 400 error with my serverless deployment for this. The only thing I would like to add on here is that serverproperties must be a Base64 encoded string.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html
My Template (that results in error):
Resources:
ServerlessMSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
ServerProperties: !Base64 |
auto.create.topics.enable=true
NB: Answering due to the fact I don't have reputation to comment.
I managed to fix the 400 issue, it looks like the AWS::MSK::Configuration resource suspects a name. While the documentation says it is not required.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html#cfn-msk-configuration-name
My template:
KafkaConfiguration:
Type: AWS::MSK::Configuration
Properties:
Name: Kafkaconfiguration
ServerProperties: |
auto.create.topics.enable=false
We are trying create a AWS::WAFv2::IPSet in our SAM template.
WhitelistedIPAddressesIPSet:
Type: AWS::WAFv2::IPSet
Properties:
Description: 'Merchant IPs'
Scope: REGIONAL
IPAddressVersion: IPV4
Addresses: [0.0.0.0/32, 0.0.10.0/32]
The creation of the IP sets is done successfully.
Once creating the AWS::WAFv2::WebACLAssociation.
WAFApiAssociation:
Type: AWS::WAFv2::WebACLAssociation
DependsOn:
- ApiGateway
- WAFWebAcl
Properties:
ResourceArn: !Sub 'arn:aws:apigateway:${AWS::Region}::/restapis/${ApiGateway}/stages/${EnvType}'
WebACLArn: !GetAtt WAFWebAcl.Arn
The CloudFormation failes and does a rollback. Error displayed is as follows:
Resource handler returned
ion message: "AWS WAF couldn?t
perform the operation
because your resource
doesn?t exist. (Service:
Wafv2, Status Code: 400,
Request ID: e337720a-e32c-
4c29-acde-1896855405c9,
Extended Request ID:
null)" (RequestToken: f24d
0488-3016-4030-3a3b-bbb246
66f130, HandlerErrorCode:
NotFound)
We tried different formatting the SAM template of the IP set, to see if that causes the issues, without any success.
Anyone that could share some helpful insights to this issue?
A) You don't need DependsOn if your resource already directly depends on those other resources. In this case it does, so you can remove this property.
B) You'll need to share your whole stack here, not just what is shared because there is likely a problem with your APIGW configuration. Because that failed to be created, it's possible you get this subsequent problem showing up.
Creating the APIGW isn't enough, you need to make sure to actually attach the WAF after the APIGW stage was created and not just the APIGW. In this case replace the ResourceArn with one that references the APIGW Stage. (And further you might need to wait for the stage deployment to finish.)
This is the APIGW template Warren Parad
CDEAPI:
Type: AWS::Serverless::Api
Properties:
# Domain:
# DomainName: !Ref CDEAPIDomainName
# SecurityPolicy: TLS_1_2
# CertificateArn: !Sub 'arn:aws:acm:us-east-1:${AWS::AccountId}:certificate/${CDEAPICertificateArn}'
# EndpointConfiguration: EDGE
# Route53:
# HostedZoneId: !Ref CDEAPIHostedZoneId
AccessLogSetting:
DestinationArn: !GetAtt CDEAPIAccessLogGroup.Arn
Format: >-
{ "requestId":"$context.requestId",
"ip":"$context.identity.sourceIp",
"caller":"$context.identity.caller",
"user":"$context.identity.user",
"userAgent":"$context.identity.userAgent",
"userArn":"$context.identity.userArn",
"requestTime":"$context.requestTime",
"requestTimeEpoch":"$context.requestTimeEpoch",
"httpMethod":"$context.httpMethod",
"resourcePath":"$context.resourcePath",
"path":"$context.path",
"status":"$context.status",
"protocol":"$context.protocol",
"responseLength":"$context.responseLength",
"responseLatency":"$context.responseLatency",
"authorizerLatency":"$context.authorizer.integrationLatency",
"integrationLatency":"$context.integrationLatency",
"integrationStatus":"$context.integrationStatus",
"xrayTraceId":"$context.xrayTraceId",
"errorMessage":"$context.error.message",
"domainName":"$context.domainName",
"domainPrefix":"$context.domainPrefix",
"tokenScopes":"$context.authorizer.claims.scope",
"tokenIat":"$context.authorizer.claims.iat",
"tokenExp":"$context.authorizer.claims.exp",
"cognitoIdentityId":"$context.identity.cognitoIdentityId",
"awsEndpointRequestId":"$context.awsEndpointRequestId",
"arn":"$context.identity.userArn",
"account":"$context.identity.accountId",
"claims-sub":"$context.authorizer.claims.sub",
"waf-error":"$context.waf.error",
"waf-status":"$context.waf.status",
"waf-latency":"$context.waf.latency",
"waf-response":"$context.waf.wafResponseCode",
"authenticate-error":"$context.authenticate.error",
"authenticate-status":"$context.authenticate.status",
"authenticate-latency":"$context.authenticate.latency",
"integration-error":"$context.integration.error",
"integration-status":"$context.integration.status",
"integration-latency":"$context.integration.latency",
"integration-requestId":"$context.integration.requestId",
"integration-integrationStatus":"$context.integration.integrationStatus",
"response-latency":"$context.responseLatency" }
StageName: !Ref EnvType
Auth:
DefaultAuthorizer: CognitoAuthorizer
AddDefaultAuthorizerToCorsPreflight: false
Authorizers:
CognitoAuthorizer:
AuthType: COGNITO_USER_POOLS
UserPoolArn: !Sub 'arn:aws:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${CognitoUserPoolArn}'
I'm trying to deploy Step Function, but I see no ways to define activity in serverless config.
AWS docs https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-stepfunctions-activity.html saying that activity should be defined that way, but every time I do sls deploy I can't see new activity in step function console. Is it possible at all to create activity via serverless or I have to run script/create it manually?
Resources:
MyActivity:
Type: "AWS::StepFunctions::Activity"
Properties:
Name: myActivity
stepFunctions:
stateMachines:
stepfunctest:
events:
- http:
path: step
method: get
definition:
Comment: "A sample application"
StartAt: extract
States:
extract:
Type: Task
Resource: "arn:aws:state:#{AWS::Region}:#{AWS::AccountId}:activity:MyActivity"
End: true
assuming you're uing the serverless plugin https://github.com/serverless-operations/serverless-step-functions. You can create the activity by adding the activity into the stepFunction
stepFuntions:
activities:
- myActivity
stateMachines:
stepfunctest:
events:
- http:
path: step
method: get
definition:
Comment: "A sample application"
StartAt: extract
States:
extract:
Type: Task
Resource: "arn:aws:state:#{AWS::Region}:#{AWS::AccountId}:activity:MyActivity"
End: true
I have two nested Cloudformation stacks - the first template needs to define a Kinesis stream, the second needs to use a reference to that stream's ARN, as an argument to a further nested stack.
So it seems I need to "export" the stream from the first template, and "import" it into the second (following AWS docs on importing/exporting values across stacks) -
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html
My export code [truncated] looks like this -
Outputs:
MyKinesisStreamOutput:
Value:
Ref: MyKinesisStream
Export:
Name: my-kinesis-stream
Resources:
MyKinesisStream:
Properties:
Name:
Ref: AppName
ShardCount: 1
Type: AWS::Kinesis::Stream
Whilst my import code [truncated] looks like this -
MyNestedStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://s3.${AWS::Region}.amazonaws.com/my-nested-stack.yaml"
Parameters:
AppName: my-nested-stack
KinesisStream:
Fn::GetAtt:
- Fn::ImportValue:
my-kinesis-stream
- Arn
But then I get the following Cloudformation error -
Template error: every Fn::GetAtt object requires two non-empty parameters, the resource name and the resource attribute
and suspect I am falling foul of this -
For the Fn::GetAtt logical resource name, you cannot use functions. You must specify a string that is a resource's logical ID.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html
Assuming I am exporting and importing the Kinesis stream correctly, how then am I supposed to get its Arn value ?
When you export something in Outputs, it's just a string, you can no longer GetAtt on it in the importing template.
What you need to do is to additionally export ARN:
Outputs:
MyKinesisStream:
Value: !Ref MyKinesisStream
Export:
Name: my-kinesis-stream
MyKinesisStreamArn:
Value: !GetAtt MyKinesisStream.Arn
Export:
Name: my-kinesis-stream-arn
I can think of two potential ways:
1) Fn::GetAtt:[!ImportValue my-kinesis-stream, Arn]
Sorry wasn't reading carefully enough
or what I would prefer
2) Directly export the required value as output: Value: !GetAtt MyKinesisStream.Arn
Hope that helps!
I want to read the URL of my database from parameter store in my CloudFormation template. This is easy enough for a single URL, but I can't figure out how to change the URL with different environments.
I have four environments (development, integration, pre-production and production) and their details are stored on four different paths in Parameter Store:
/database/dev/url
/database/int/url
/database/ppe/url
/database/prod/url
I now want to pick the correct Database URL when deploying via CloudFormation. How can I do this?
Parameters:
Environment:
Type: String
Default: dev
AllowedValues:
- dev
- int
- ppe
- prod
DatabaseUrl:
Type: 'AWS::SSM::Parameter::Value<String>'
# Obviously the '+' operator here won't work - so what do I do?
Default: '/database/' + Environment + '/url'
This feature isn't as neat as one would wish. You have to actually pass name/path of each parameter that you want to look up from the parameter store.
Template:
AWSTemplateFormatVersion: 2010-09-09
Description: Example
Parameters:
BucketNameSuffix:
Type: AWS::SSM::Parameter::Value<String>
Default: /example/dev/BucketNameSuffix
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub parameter-store-example-${BucketNameSuffix}
If you don't pass any parameters to the template, BucketNameSuffix will be populated with the value stored at /example/dev/BucketNameSuffix. If you want to use, say, a prod value (pointed to by /example/prod/BucketNameSuffix), then you should specify value for parameter BucketNameSuffix, but instead of passing the actual value you should pass the alternative name of the parameter to use, so you'd pass /example/prod/BucketNameSuffix.
aws cloudformation update-stack --stack-name example-dev \
--template-body file://./example-stack.yml
aws cloudformation update-stack --stack-name example-prod \
--template-body file://./example-stack.yml \
--parameters ParameterKey=BucketNameSuffix,ParameterValue=/example/prod/BucketNameSuffix
A not so great AWS blog post on this: https://aws.amazon.com/blogs/mt/integrating-aws-cloudformation-with-aws-systems-manager-parameter-store/
Because passing million meaningless parameters seems stupid, I might actually generate an environment-specific template and set the right Default: in the generated template so for prod environment Default would be /example/prod/BucketNameSuffix and then I can update the prod stack without passing any parameters.
You can populate CloudFormation templates with parameters stored in AWS Systems Manager Parameter Store using Dynamic References
In this contrived example we make two lookups using resolve:ssm and replace the environment using !Join and !Sub
AWSTemplateFormatVersion: 2010-09-09
Parameters:
Environment:
Type: String
Default: prod
AllowedValues:
- prod
- staging
Resources:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
Image: !Join
- ''
- - 'docker.io/bitnami/redis#'
- !Sub '{{resolve:ssm:/app/${Environment}/digest-sha/redis}}'
Environment:
- Name: DB_HOST
Value: !Sub '{{resolve:ssm:/app/${Environment}/db-host}}'
You can make use of Fn::Join here.
Here is some pseudo code.
You will have to take Environment as an parameter, which you are already doing.
Creating the required string in the resources where DatabaseUrl is required.
Resources :
Instance :
Type : 'AWS::Some::Resource'
Properties :
DatabaseURL : !Join [ "", [ "/database/", !Ref "Environment" , "/url ] ]
Hope this helps.
Note: You cannot assign value to a Parameter dynamically using some computation logic. All the values for defined parameters should be given as Input.
I like Fn::Sub, which is much cleaner and easy to read.
!Sub "/database/${Environment}/url"
I am stuck into the same problem, below are my findings:
We can compose values and descriptions while writing into the SSM Parameter Store from CloudFormation like below :
LambdaARN:
Type: AWS::SSM::Parameter
Properties:
Type: String
Description: !Sub "Lambda ARN from ${AWS::StackName}"
Name: !Sub "/secure/${InstallationId}/${AWS::StackName}/lambda-function-arn"
Value: !GetAtt LambdaFunction.Arn
We can not compose values/defaults to looking for in SSM Parameter Store. Like below:
Parameters:
...
LambdaARN:
Type: Type: AWS::SSM::Parameter::Value<String>
Value: !Sub "/secure/${InstallationId}/teststack/lambda-function-arn"
This is not allowed as per AWS Documentation[1]. Both(Sub/Join) Functions won't work. Following is the error which I was getting:
An error occurred (ValidationError) when calling the CreateChangeSet
operation: Template format error: Every Default member must be a
string.
Composing and passing values while creating stacks can be done like this:
Parameters:
...
LambdaARN:
Type: Type: AWS::SSM::Parameter::Value<String>
....
$ aws cloudformation deploy --template-file cfn.yml --stack-name mystack --parameter-overrides 'LambdaARN=/secure/devtest/teststack/lambda_function-arn'
If you add your custom tags while putting the values in the Parameter Store, it will overwrite the default tags added by CFN.
Default Tags:
- aws:cloudformation:stack-id
- aws:cloudformation:stack-name
- aws:cloudformation:logical-id
Every time we update the values in parameter store, it creates a new version, which is beneficial when we are using DynamicResolvers, this can serve as a solution to the problem in the question like
{{ resolve:ssm:/my/value:1}}
The last field is the version. Where different versions can point to different environments.
We are using versions with the parameters, and adding the labels to them[2], this can't be done via CFN[3], only possible way via CLI or AWS Console. This is AWS's way of handling multiple environments.
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html
[2] https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-labels.html
[3] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ssm-parameter.html