I'm trying to create a AWS::MSK::Configuration resource, as described here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html.
This is my CF template:
Resources:
MSKConfig:
Type: AWS::MSK::Configuration
Properties:
Description: Basic configuration.
Name: test-msk-configuration
ServerProperties: |
auto.create.topics.enable=false
........
It doesn't work failing with the following error:
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: #Unrecognized resource types: [AWS::MSK::Configuration]
I'm not able to find any example with this resource type online. Has anyone ever used it?
Update on 26/03/2022:
I was able to get this deployed. I use the serverless framework for deployment.
MSK-Cluster.yml:
Resources:
ServerlessMSK:
Type: AWS::MSK::Cluster
Properties:
ClusterName: ${self:service}-${self:provider.stage}-msk
KafkaVersion: 2.6.2
BrokerNodeGroupInfo:
InstanceType: kafka.t3.small
ClientSubnets:
- !Ref ServerlessPrivateSubnet1
- !Ref ServerlessPrivateSubnet2
- !Ref ServerlessPrivateSubnet3
SecurityGroups:
- !GetAtt ServerlessMSKSecurityGroup.GroupId
StorageInfo:
EBSStorageInfo:
VolumeSize: 10
NumberOfBrokerNodes: 3
EncryptionInfo:
EncryptionInTransit:
ClientBroker: TLS
InCluster: true
EnhancedMonitoring: PER_TOPIC_PER_BROKER
ConfigurationInfo:
Arn: !GetAtt ServerlessMSKConfiguration.Arn
Revision: 1
MSK-config.yml
Resources:
ServerlessMSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
Description: cluster for msk cluster-${sls:stage}
Name: node-mongo-kafka-experiment-${sls:stage}-config
ServerProperties: ${file('./assets/server.properties')}
server.properties
auto.create.topics.enable=true
default.replication.factor=2
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=10
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000
basically, no base64 was used. I just referenced the file in the deployment and managed to get it up. Hope this helps.
Orginal Answer Below:
I haven't been able to deploy this correctly too. But maybe I can point you in the right direction.I always get back a 400 error with my serverless deployment for this. The only thing I would like to add on here is that serverproperties must be a Base64 encoded string.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html
My Template (that results in error):
Resources:
ServerlessMSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
ServerProperties: !Base64 |
auto.create.topics.enable=true
NB: Answering due to the fact I don't have reputation to comment.
I managed to fix the 400 issue, it looks like the AWS::MSK::Configuration resource suspects a name. While the documentation says it is not required.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html#cfn-msk-configuration-name
My template:
KafkaConfiguration:
Type: AWS::MSK::Configuration
Properties:
Name: Kafkaconfiguration
ServerProperties: |
auto.create.topics.enable=false
We are trying create a AWS::WAFv2::IPSet in our SAM template.
WhitelistedIPAddressesIPSet:
Type: AWS::WAFv2::IPSet
Properties:
Description: 'Merchant IPs'
Scope: REGIONAL
IPAddressVersion: IPV4
Addresses: [0.0.0.0/32, 0.0.10.0/32]
The creation of the IP sets is done successfully.
Once creating the AWS::WAFv2::WebACLAssociation.
WAFApiAssociation:
Type: AWS::WAFv2::WebACLAssociation
DependsOn:
- ApiGateway
- WAFWebAcl
Properties:
ResourceArn: !Sub 'arn:aws:apigateway:${AWS::Region}::/restapis/${ApiGateway}/stages/${EnvType}'
WebACLArn: !GetAtt WAFWebAcl.Arn
The CloudFormation failes and does a rollback. Error displayed is as follows:
Resource handler returned
ion message: "AWS WAF couldn?t
perform the operation
because your resource
doesn?t exist. (Service:
Wafv2, Status Code: 400,
Request ID: e337720a-e32c-
4c29-acde-1896855405c9,
Extended Request ID:
null)" (RequestToken: f24d
0488-3016-4030-3a3b-bbb246
66f130, HandlerErrorCode:
NotFound)
We tried different formatting the SAM template of the IP set, to see if that causes the issues, without any success.
Anyone that could share some helpful insights to this issue?
A) You don't need DependsOn if your resource already directly depends on those other resources. In this case it does, so you can remove this property.
B) You'll need to share your whole stack here, not just what is shared because there is likely a problem with your APIGW configuration. Because that failed to be created, it's possible you get this subsequent problem showing up.
Creating the APIGW isn't enough, you need to make sure to actually attach the WAF after the APIGW stage was created and not just the APIGW. In this case replace the ResourceArn with one that references the APIGW Stage. (And further you might need to wait for the stage deployment to finish.)
This is the APIGW template Warren Parad
CDEAPI:
Type: AWS::Serverless::Api
Properties:
# Domain:
# DomainName: !Ref CDEAPIDomainName
# SecurityPolicy: TLS_1_2
# CertificateArn: !Sub 'arn:aws:acm:us-east-1:${AWS::AccountId}:certificate/${CDEAPICertificateArn}'
# EndpointConfiguration: EDGE
# Route53:
# HostedZoneId: !Ref CDEAPIHostedZoneId
AccessLogSetting:
DestinationArn: !GetAtt CDEAPIAccessLogGroup.Arn
Format: >-
{ "requestId":"$context.requestId",
"ip":"$context.identity.sourceIp",
"caller":"$context.identity.caller",
"user":"$context.identity.user",
"userAgent":"$context.identity.userAgent",
"userArn":"$context.identity.userArn",
"requestTime":"$context.requestTime",
"requestTimeEpoch":"$context.requestTimeEpoch",
"httpMethod":"$context.httpMethod",
"resourcePath":"$context.resourcePath",
"path":"$context.path",
"status":"$context.status",
"protocol":"$context.protocol",
"responseLength":"$context.responseLength",
"responseLatency":"$context.responseLatency",
"authorizerLatency":"$context.authorizer.integrationLatency",
"integrationLatency":"$context.integrationLatency",
"integrationStatus":"$context.integrationStatus",
"xrayTraceId":"$context.xrayTraceId",
"errorMessage":"$context.error.message",
"domainName":"$context.domainName",
"domainPrefix":"$context.domainPrefix",
"tokenScopes":"$context.authorizer.claims.scope",
"tokenIat":"$context.authorizer.claims.iat",
"tokenExp":"$context.authorizer.claims.exp",
"cognitoIdentityId":"$context.identity.cognitoIdentityId",
"awsEndpointRequestId":"$context.awsEndpointRequestId",
"arn":"$context.identity.userArn",
"account":"$context.identity.accountId",
"claims-sub":"$context.authorizer.claims.sub",
"waf-error":"$context.waf.error",
"waf-status":"$context.waf.status",
"waf-latency":"$context.waf.latency",
"waf-response":"$context.waf.wafResponseCode",
"authenticate-error":"$context.authenticate.error",
"authenticate-status":"$context.authenticate.status",
"authenticate-latency":"$context.authenticate.latency",
"integration-error":"$context.integration.error",
"integration-status":"$context.integration.status",
"integration-latency":"$context.integration.latency",
"integration-requestId":"$context.integration.requestId",
"integration-integrationStatus":"$context.integration.integrationStatus",
"response-latency":"$context.responseLatency" }
StageName: !Ref EnvType
Auth:
DefaultAuthorizer: CognitoAuthorizer
AddDefaultAuthorizerToCorsPreflight: false
Authorizers:
CognitoAuthorizer:
AuthType: COGNITO_USER_POOLS
UserPoolArn: !Sub 'arn:aws:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${CognitoUserPoolArn}'
I am working on a proof of concept for deploying a conformance pack via AWS cloudformation and I am stumped by the error "Parameter values specified for a template which does not require them." The config rule I am using does require a parameter. Code is attached. I have also tested the template with cfn-lint and do not receive any feedback/errors.
My template is "simple" and below:
Parameters:
ElbPredefinedSecurityPolicySslCheckParamPredefinedPolicyName:
Default: ELBSecurityPolicy-2016-08
Type: String
Resources:
TestingConformancePack:
Type: AWS::Config::ConformancePack
Properties:
ConformancePackName: TestCP
ConformancePackInputParameters:
-
ParameterName: PredefinedPolicyName
ParameterValue: !Ref ElbPredefinedSecurityPolicySslCheckParamPredefinedPolicyName
TemplateBody: |
Resources:
ElbPredefinedSecurityPolicySslCheck:
Properties:
ConfigRuleName: elb-predefined-security-policy-ssl-check
InputParameters:
predefinedPolicyName:
Ref: ElbPredefinedSecurityPolicySslCheckParamPredefinedPolicyName
Scope:
ComplianceResourceTypes:
- AWS::ElasticLoadBalancing::LoadBalancer
Source:
Owner: AWS
SourceIdentifier: ELB_PREDEFINED_SECURITY_POLICY_SSL_CHECK
Type: AWS::Config::ConfigRule
The cause is that you are passing a parameter (the one specified in ConformancePackInputParameters) to a CloudFormation template (the one specified in TemplateBody) that does not contain a Parameters section and therefore expects no parameters. To solve this, you need to add a parameter to the inner CloudFormation template, which you can then refer to in predefinedPolicyName:
The following template works for me:
Parameters:
ElbPredefinedSecurityPolicySslCheckParamPredefinedPolicyName:
Default: ELBSecurityPolicy-2016-08
Type: String
Resources:
TestingConformancePack:
Type: AWS::Config::ConformancePack
Properties:
ConformancePackName: TestCP
ConformancePackInputParameters:
-
ParameterName: PredefinedPolicyName
ParameterValue: !Ref ElbPredefinedSecurityPolicySslCheckParamPredefinedPolicyName
TemplateBody: |
Parameters:
PredefinedPolicyName:
Type: String
Resources:
ElbPredefinedSecurityPolicySslCheck:
Properties:
ConfigRuleName: elb-predefined-security-policy-ssl-check
InputParameters:
predefinedPolicyName:
Ref: PredefinedPolicyName
Scope:
ComplianceResourceTypes:
- AWS::ElasticLoadBalancing::LoadBalancer
Source:
Owner: AWS
SourceIdentifier: ELB_PREDEFINED_SECURITY_POLICY_SSL_CHECK
Type: AWS::Config::ConfigRule
I was making a test case resource using cloudformation and stumbled upon this same error.
“Parameter values specified for a template which does not require them.”
Since it was a test case, I didn't use any parameters at all. The above answer was helpful for me to understand it has to do something with parameters. Even though you are not using any, there are some parameters passed while deploying the cfn.
By default, cloudformation also sends env as parameter which needs to come under parameters as such. (Below code snippet in JSON)
"Parameters": {
"env": {
"Type": "String"
}
},
Hope this was helpful.
I am writing a cloudformation template for creating an aws step function and statemachine. Following is the part of my template which is causing the error
AWSTemplateFormatVersion: 2010-09-09
Transform:
- StepFunctionsYamlTransform
StepFunctionsStateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
StateMachineName: MyStack
RoleArn: !GetAtt StateMachineRole.Arn
DefinitionStringYaml: !Sub
- |
Comment: My-Stack-workflow
StartAt: LambdaToStart
TimeoutSeconds: 43200
States:
LambdaToStart:
Type: Task
Resource: "${LambdaToStartArn}"
Next: WaitToWriteInS3
WaitToWriteInS3:
Type: Wait
Seconds: 5
Next: Batch_Job_1
Batch_Job_1:
Type: Task
Next: LambdaForTriggerEmrJob
Resource: arn:aws:states:::batch:submitJob.sync
Parameters:
JobName: "${BatchJob1}"
JobDefinition: "${BatchJob1DefinitionArn}"
JobQueue: arn:aws:batch:${AWS::Region}:${AWS::AccountId}:job-queue/${QueueName}
LambdaForTriggerEmrJob:
Type: Task
Resource: "${LambdaForEmrArn}"
Next: WaitFoEmrState
WaitFoEmrState:
Type: Wait
Seconds: 90
Next: CheckEmrState
CheckEmrState:
Type: Task
Resource: "${ClusterStateCheckArn}"
InputPath: "$.input.cluster" # Values coming from lambda
ResultPath: "$.input.cluster" # Values coming from lambda
Retry: *LambdaRetryConfig
Next: IsClusterRunning
IsClusterRunning:
Type: Choice
Default: WaitFoEmrState
Choices:
- Variable: "$.input.cluster.state"
StringEquals: FAILED
Next: StateMachineFailure
- Variable: "$.input.cluster.state" # Values coming from lambda
StringEquals: SUCCEEDED
Next: FinalBatchJob
StateMachineFailure:
Type: Fail
FinalBatchJob:
Type: Task
Resource: arn:aws:states:::batch:submitJob.sync
Parameters:
JobName: "${FinalBatch}"
JobDefinition: "${FinalBatchDefinitionArn}"
JobQueue: arn:aws:batch:${AWS::Region}:${AWS::AccountId}:job-queue/${QueueName
End: true
- LambdaToStartArn: !GetAtt LambdaToStart.Arn
LambdaForEmrArn: !GetAtt LambdaForEmr.Arn
BatchJob1DefinitionArn: !Ref BatchJob1Definition
FinalBatchDefinitionArn: !Ref FinalBatchDefinition
BatchJob1: !Sub ${AWS::StackName}-batch-1
FinalBatch: !Sub ${AWS::StackName}-final-batch
ClusterStateCheckArn: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:cluster-state
It returns the following error
Failed to create the changeset: Waiter ChangeSetCreateComplete failed:
Waiter encountered a terminal failure state Status: FAILED. Reason:
Transform {AWS::AccountId}::StepFunctionsYamlTransform failed without an
error message.
Can anyone help in recognising the solution to this? I can't debug a lot since it fails without an error message. TIA
AWS cloudformation errors are sometimes quite wierd and its difficult to debug them. But I found the error. It was 9th line JobQueue: arn:aws:batch:${AWS::Region}:${AWS::AccountId}:job-queue/${QueueName and one can easily see that I missed } at the end. So it was a syntax error
I am using the aws eb deploy command on ubuntu to deploy a cloudformation script shown below. I get an error shown below.
Note: My other cloudformation scripts work without any problems.
Error Invalid Yaml: mapping values are not allowed here in ""
CacheSecurityGroupName: Ref: "CacheSecurityGroup"
^, JSON exception: Invalid JSON: Unexpected character (R) at position 0.. Update the configuration file.
ERROR: Failed to deploy application.
Resources:
CacheSecurityGroupIngress:
Type: "AWS::ElastiCache::SecurityGroupIngress"
Properties:
CacheSecurityGroupName: Ref: "CacheSecurityGroup"
EC2SecurityGroupName: Ref: "AWSEBSecurityGroup"
Looking for pointers to solve the issue
You should either use the full Ref function form in a new line, like this:
Resources:
CacheSecurityGroupIngress:
Type: "AWS::ElastiCache::SecurityGroupIngress"
Properties:
CacheSecurityGroupName:
Ref: "CacheSecurityGroup"
EC2SecurityGroupName:
Ref: "AWSEBSecurityGroup"
...or the short form, like this:
Resources:
CacheSecurityGroupIngress:
Type: "AWS::ElastiCache::SecurityGroupIngress"
Properties:
CacheSecurityGroupName: !Ref "CacheSecurityGroup"
EC2SecurityGroupName: !Ref "AWSEBSecurityGroup"