I'm trying to create a cloudFormation stack with MSK Configuration and associating MSK Configuration with MSK Cluster. Creation of AWS::MSK::Configuration returns only ARN while I need ARN and Revision number to associate MSK Configuration to MSK Cluster. Is there any way to achieve this? Currently I'm hard-coding it to 1 which means it will work only for creating stack.
...
MSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
Name: aws-msk-configuration
ServerProperties: |
auto.create.topics.enable = true
zookeeper.connection.timeout.ms = 1000
log.roll.ms = 604800000
MSKCluster:
Type: AWS::MSK::Cluster
Properties:
ClusterName: !Ref ClusterName
ClientAuthentication: !If
- UsingIamAuthentication
- Sasl:
Iam:
Enabled: true
- Sasl:
Scram:
Enabled: true
ConfigurationInfo:
Arn: !GetAtt MSKConfiguration.Arn
Revision: 1
...
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html
You can only get the latest version if you define a custom resource. Since you program the full logic of the resource, you can do what you want, including automatically setting up latest version for MKS.
Related
I want to trigger a lambda whenever a new EC2 instance is registred in SSM's Fleet Manager (meaning the instance can be connected to using SSM), however I can't find what pattern to use in EventBridge.
Within EventBridge, I tried using the following pattern I found in the docs (so far its looks like the closest thing to my goal):
{
"source": ["aws.ssm"],
"detail-type": ["Inventory Resource State Change"]
}
However when I create a new EC2 and wait for its SSM agent to become active, it still doesn't trigger the above pattern.
Any idea how to catch this kind of event?
I think you have to go through CloudTrail API call.
Please find below a CloudFormation template I used in the past that was working. Please note that it just provides the SSM resources. You need to add your own SQS queue as well (see SQS.ARN) and I've used the association with the tag registration set to enabled. So that if you have a lambda function connected, you can set it to false so if the instance connect again, it won't go to the same process again.
AWSTemplateFormatVersion: "2010-09-09"
Description: >
SSM Registration event
# Description of the resources to be created.
Resources:
RegistrationDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: Command
Content:
schemaVersion: "2.2"
description: >
An Automation Document ran by registered instances that gathers their software inventory
and automatically updates their AWS SSM Agent to the latest version.
mainSteps:
- name: GatherSoftware
action: aws:softwareInventory
- name: Sleep
action: aws:runShellScript
inputs:
runCommand:
- sleep 20 || true
- name: UpdateAgent
action: aws:updateSsmAgent
inputs:
agentName: amazon-ssm-agent
source: https://s3.{Region}.amazonaws.com/amazon-ssm-{Region}/ssm-agent-manifest.json
allowDowngrade: "false"
RegistrationDocumentAssociation:
Type: AWS::SSM::Association
Properties:
AssociationName: !Sub registration-association-${AWS::StackName}
Name: !Ref RegistrationDocument
Targets:
- Key: tag:registration
Values:
- enabled
RegistrationEventRule:
Type: AWS::Events::Rule
Properties:
Description: >
Events Rule that monitors registration of AWS SSM instances
and logs them to an SQS queue.
EventPattern:
source:
- aws.ssm
detail-type:
- AWS API Call via CloudTrail
detail:
eventName:
- UpdateInstanceAssociationStatus
requestParameters:
associationId:
- !Ref RegistrationDocumentAssociation
executionResult:
status:
- Success
State: ENABLED
Targets:
- Arn: SQS.ARN
Id: SqsRegistrationSubscription
SqsParameters:
MessageGroupId: registration.events
I'm trying to create a AWS::MSK::Configuration resource, as described here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html.
This is my CF template:
Resources:
MSKConfig:
Type: AWS::MSK::Configuration
Properties:
Description: Basic configuration.
Name: test-msk-configuration
ServerProperties: |
auto.create.topics.enable=false
........
It doesn't work failing with the following error:
An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: #Unrecognized resource types: [AWS::MSK::Configuration]
I'm not able to find any example with this resource type online. Has anyone ever used it?
Update on 26/03/2022:
I was able to get this deployed. I use the serverless framework for deployment.
MSK-Cluster.yml:
Resources:
ServerlessMSK:
Type: AWS::MSK::Cluster
Properties:
ClusterName: ${self:service}-${self:provider.stage}-msk
KafkaVersion: 2.6.2
BrokerNodeGroupInfo:
InstanceType: kafka.t3.small
ClientSubnets:
- !Ref ServerlessPrivateSubnet1
- !Ref ServerlessPrivateSubnet2
- !Ref ServerlessPrivateSubnet3
SecurityGroups:
- !GetAtt ServerlessMSKSecurityGroup.GroupId
StorageInfo:
EBSStorageInfo:
VolumeSize: 10
NumberOfBrokerNodes: 3
EncryptionInfo:
EncryptionInTransit:
ClientBroker: TLS
InCluster: true
EnhancedMonitoring: PER_TOPIC_PER_BROKER
ConfigurationInfo:
Arn: !GetAtt ServerlessMSKConfiguration.Arn
Revision: 1
MSK-config.yml
Resources:
ServerlessMSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
Description: cluster for msk cluster-${sls:stage}
Name: node-mongo-kafka-experiment-${sls:stage}-config
ServerProperties: ${file('./assets/server.properties')}
server.properties
auto.create.topics.enable=true
default.replication.factor=2
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=10
num.replica.fetchers=2
replica.lag.time.max.ms=30000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
unclean.leader.election.enable=true
zookeeper.session.timeout.ms=18000
basically, no base64 was used. I just referenced the file in the deployment and managed to get it up. Hope this helps.
Orginal Answer Below:
I haven't been able to deploy this correctly too. But maybe I can point you in the right direction.I always get back a 400 error with my serverless deployment for this. The only thing I would like to add on here is that serverproperties must be a Base64 encoded string.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html
My Template (that results in error):
Resources:
ServerlessMSKConfiguration:
Type: AWS::MSK::Configuration
Properties:
ServerProperties: !Base64 |
auto.create.topics.enable=true
NB: Answering due to the fact I don't have reputation to comment.
I managed to fix the 400 issue, it looks like the AWS::MSK::Configuration resource suspects a name. While the documentation says it is not required.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html#cfn-msk-configuration-name
My template:
KafkaConfiguration:
Type: AWS::MSK::Configuration
Properties:
Name: Kafkaconfiguration
ServerProperties: |
auto.create.topics.enable=false
The below CF template is indented to create a stack set containing one role with specific parameter values depending on the OU where the account is located. So if the account is located / created in OU A the trust relationship of the debug role must point to account "111111111111".
AWSTemplateFormatVersion: "2010-09-09"
Description: Debug Role
Resources:
StackSet:
Type: AWS::CloudFormation::StackSet
Properties:
Description: Debug Role Stack Set
PermissionModel: SERVICE_MANAGED
Capabilities:
- CAPABILITY_NAMED_IAM
AutoDeployment:
Enabled: true
RetainStacksOnAccountRemoval: false
Parameters:
- ParameterKey: TrustAccountId
ParameterValue: "123456789123" # default value
OperationPreferences:
RegionConcurrencyType: PARALLEL
FailureToleranceCount: 0
MaxConcurrentPercentage: 100
StackInstancesGroup:
- DeploymentTargets:
OrganizationalUnitIds:
- ou-aaaa # A
ParameterOverrides:
- ParameterKey: "TrustAccountId"
ParameterValue: "111111111111"
Regions:
- 'eu-central-1'
- DeploymentTargets:
OrganizationalUnitIds:
- ou-bbbb # B
ParameterOverrides:
- ParameterKey: "TrustAccountId"
ParameterValue: "222222222222"
Regions:
- 'eu-central-1'
StackSetName: debug-role-stack-set
TemplateBody: |
AWSTemplateFormatVersion: 2010-09-09
Description: Debug Role
Parameters:
TrustAccountId:
Description: Trust to account
Type: String
AllowedPattern: \d{12}
Resources:
DebugRole:
Type: AWS::IAM::Role
Description: "Role for debugging accounts"
Properties:
RoleName: DebugRole
Path: /
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS:
- Ref: TrustAccountId
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
Now the issue is that all account within the stack instances group, regardless if existing or new account, get the stack applied with the default parameter value "123456789123". So all accounts get the role having the incorrect trust relationship to "123456789123".
The only way to apply the correct value based on the OU is to update the stack instances with the CLI.
aws cloudformation update-stack-instances --stack-set-name debug-role-stack-set --deployment-targets OrganizationalUnitIds=ou-aaaa --regions eu-central-1 --operation-preferences FailureToleranceCount=0,MaxConcurrentCount=5 --parameter-overrides Pa
rameterKey=TrustAccountId,ParameterValue=111111111111
Is there any way to apply the correct parameter values during the stack creation without updating them afterwards? Because for every new account all stack set instances needs to be updated.
Since the stack instances have not been executed with the correct parameter value, I ended up having one stack set for each OU. Of course, this solution is only feasible for static environments. So in my case, if new OUs pop up new stack sets must be created.
We are trying create a AWS::WAFv2::IPSet in our SAM template.
WhitelistedIPAddressesIPSet:
Type: AWS::WAFv2::IPSet
Properties:
Description: 'Merchant IPs'
Scope: REGIONAL
IPAddressVersion: IPV4
Addresses: [0.0.0.0/32, 0.0.10.0/32]
The creation of the IP sets is done successfully.
Once creating the AWS::WAFv2::WebACLAssociation.
WAFApiAssociation:
Type: AWS::WAFv2::WebACLAssociation
DependsOn:
- ApiGateway
- WAFWebAcl
Properties:
ResourceArn: !Sub 'arn:aws:apigateway:${AWS::Region}::/restapis/${ApiGateway}/stages/${EnvType}'
WebACLArn: !GetAtt WAFWebAcl.Arn
The CloudFormation failes and does a rollback. Error displayed is as follows:
Resource handler returned
ion message: "AWS WAF couldn?t
perform the operation
because your resource
doesn?t exist. (Service:
Wafv2, Status Code: 400,
Request ID: e337720a-e32c-
4c29-acde-1896855405c9,
Extended Request ID:
null)" (RequestToken: f24d
0488-3016-4030-3a3b-bbb246
66f130, HandlerErrorCode:
NotFound)
We tried different formatting the SAM template of the IP set, to see if that causes the issues, without any success.
Anyone that could share some helpful insights to this issue?
A) You don't need DependsOn if your resource already directly depends on those other resources. In this case it does, so you can remove this property.
B) You'll need to share your whole stack here, not just what is shared because there is likely a problem with your APIGW configuration. Because that failed to be created, it's possible you get this subsequent problem showing up.
Creating the APIGW isn't enough, you need to make sure to actually attach the WAF after the APIGW stage was created and not just the APIGW. In this case replace the ResourceArn with one that references the APIGW Stage. (And further you might need to wait for the stage deployment to finish.)
This is the APIGW template Warren Parad
CDEAPI:
Type: AWS::Serverless::Api
Properties:
# Domain:
# DomainName: !Ref CDEAPIDomainName
# SecurityPolicy: TLS_1_2
# CertificateArn: !Sub 'arn:aws:acm:us-east-1:${AWS::AccountId}:certificate/${CDEAPICertificateArn}'
# EndpointConfiguration: EDGE
# Route53:
# HostedZoneId: !Ref CDEAPIHostedZoneId
AccessLogSetting:
DestinationArn: !GetAtt CDEAPIAccessLogGroup.Arn
Format: >-
{ "requestId":"$context.requestId",
"ip":"$context.identity.sourceIp",
"caller":"$context.identity.caller",
"user":"$context.identity.user",
"userAgent":"$context.identity.userAgent",
"userArn":"$context.identity.userArn",
"requestTime":"$context.requestTime",
"requestTimeEpoch":"$context.requestTimeEpoch",
"httpMethod":"$context.httpMethod",
"resourcePath":"$context.resourcePath",
"path":"$context.path",
"status":"$context.status",
"protocol":"$context.protocol",
"responseLength":"$context.responseLength",
"responseLatency":"$context.responseLatency",
"authorizerLatency":"$context.authorizer.integrationLatency",
"integrationLatency":"$context.integrationLatency",
"integrationStatus":"$context.integrationStatus",
"xrayTraceId":"$context.xrayTraceId",
"errorMessage":"$context.error.message",
"domainName":"$context.domainName",
"domainPrefix":"$context.domainPrefix",
"tokenScopes":"$context.authorizer.claims.scope",
"tokenIat":"$context.authorizer.claims.iat",
"tokenExp":"$context.authorizer.claims.exp",
"cognitoIdentityId":"$context.identity.cognitoIdentityId",
"awsEndpointRequestId":"$context.awsEndpointRequestId",
"arn":"$context.identity.userArn",
"account":"$context.identity.accountId",
"claims-sub":"$context.authorizer.claims.sub",
"waf-error":"$context.waf.error",
"waf-status":"$context.waf.status",
"waf-latency":"$context.waf.latency",
"waf-response":"$context.waf.wafResponseCode",
"authenticate-error":"$context.authenticate.error",
"authenticate-status":"$context.authenticate.status",
"authenticate-latency":"$context.authenticate.latency",
"integration-error":"$context.integration.error",
"integration-status":"$context.integration.status",
"integration-latency":"$context.integration.latency",
"integration-requestId":"$context.integration.requestId",
"integration-integrationStatus":"$context.integration.integrationStatus",
"response-latency":"$context.responseLatency" }
StageName: !Ref EnvType
Auth:
DefaultAuthorizer: CognitoAuthorizer
AddDefaultAuthorizerToCorsPreflight: false
Authorizers:
CognitoAuthorizer:
AuthType: COGNITO_USER_POOLS
UserPoolArn: !Sub 'arn:aws:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${CognitoUserPoolArn}'
I Have a parameter PreferredMaintenanceWindow setup within AWS::RDS::DBCluster in my cloudformation template and it was all good.
Recently I had to add the parameter EnableCloudwatchLogsExport to the template, but when I tried to deploy the stack it failed with the following message:
You currently can't modify MaintenanceWindow with Aurora Serverless.
(Service: AmazonRDS; Status Code: 400; Error Code:
InvalidParameterCombination; Request ID:
bf40fd7e-e599-481a-ac23-b3c68e9794a1)
This is when I didn't even touch that parameter, did anyone faced such issue or any workaround which I can use with CloudFormation
PS: I am aware that I can setup the EnableCloudwatchLogsExport from the CLI but I don't want drift and I would really want it to do it via cloudformatio.
Update:
RDSServerlessDB:
Type: "AWS::RDS::DBCluster"
Properties:
AvailabilityZones:
Fn::GetAZs: !Ref 'AWS::Region'
BackupRetentionPeriod: 3 # Number of days
DatabaseName: !Join ["_", [!Join ["_", !Split [ "-", !Sub '${MasterStack}']], 'fol']]
DBClusterParameterGroupName: "default.aurora5.6"
DBSubnetGroupName: !Ref RDSDBSubnetGroup
EnableCloudwatchLogsExports:
- error
- general
- slowquery
- audit
Engine: "aurora"
EngineMode: "serverless"
# EngineVersion: "5.6.10a"
MasterUsername: "username"
MasterUserPassword: "{{resolve:ssm-secure:/password:1}}"
# Port: 3306
# PreferredMaintenanceWindow: "sun:01:00-sun:03:00"
ScalingConfiguration:
AutoPause: True
MaxCapacity: 32
MinCapacity: 2
SecondsUntilAutoPause: 300
Tags:
- Key: owner
Value: !Ref StackOwner
- Key: task
Value: !Ref Task
VpcSecurityGroupIds:
- Ref: RDSSecurityGroup
It's a bug.
As a work-around, you can temporarily remove (or comment out) PreferredBackupWindow and PreferredMaintenanceWindow from the template. In my testing, the CF update succeeds and both values are left intact on the instance configuration.
There is actually some debate as to what the expected behavior is. The bug report comments say these settings are not supported in serverless mode, but the AWS Documentation and AWS Console don't seem to support that assertion.
Apparently this is a limitation (among many others) of Aurora Serverless v1: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html#aurora-serverless.limitations (it's the last in the list of limitations).
I tried setting it with the AWS CLI v1 and with the AWS Java SDK v2:
software.amazon.awssdk.services.rds.model.ScalingConfiguration scalingConfiguration = software.amazon.awssdk.services.rds.model.ScalingConfiguration.builder().minCapacity(1).build()
final ModifyDbClusterRequest request = ModifyDbClusterRequest.builder().dbClusterIdentifier(clusterIdentifier).preferredBackupWindow("18:00-19:00").scalingConfiguration(scalingConfiguration).build()
final ModifyDbClusterResponse response = rdsClient.modifyDBCluster(request)
It returns the same error regardless.