I am trying to create 2 SQS using a single cloudformation template. I want to apply same Queue policy to both the queues. Currently I have the below template to create 1 queue and its corresponding dlq. Do I have to write all the resources twice or can I reuse the same queue policy for Queue2?
Resources:
Queue1:
Type: AWS::SQS::Queue
Properties:
RedrivePolicy:
deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
maxReceiveCount: 10
QueueName: queue1
MessageRetentionPeriod: 604800
VisibilityTimeout: 150
ReceiveMessageWaitTimeSeconds: 0
DeadLetterQueue1:
Type: AWS::SQS::Queue
Properties:
QueueName: queue1-dlq
MessageRetentionPeriod: 1209600
QueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: allow-service-to-read-and-write-messages
Effect: Allow
Principal:
AWS:
- !Sub arn:aws:iam::${AWS::AccountId}:role/User
Resource: !GetAtt Queue1.Arn
Action:
- SQS:ChangeMessageVisibility
- SQS:DeleteMessage
- SQS:GetQueueUrl
- SQS:ReceiveMessage
- SQS:SendMessage
Queues:
- !Ref Queue1
DeadLetterQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: allow-service-to-read-messages
Effect: Allow
Principal:
AWS:
- !Sub arn:aws:iam::${AWS::AccountId}:role/User
Resource: !GetAtt DeadLetterQueue1.Arn
Action:
- SQS:ChangeMessageVisibility
- SQS:DeleteMessage
- SQS:GetQueueUrl
- SQS:ReceiveMessage
- Sid: allow-queue-to-send-messages
Effect: Allow
Principal: '*'
Resource: !GetAtt DeadLetterQueue1.Arn
Action:
- SQS:SendMessage
Condition:
ArnEquals:
"aws:SourceArn": !Ref Queue1
Queues:
- !Ref DeadLetterQueue1
Outputs:
SourceQueueURL:
Description: URL of source queue
Value:
Ref: Queue1
SourceQueueARN:
Value: !GetAtt Queue1.Arn
Description: Arn of created SQS
DeadLetterQueueURL:
Description: URL of dead-letter queue
Value:
Ref: DeadLetterQueue1
DeadLetterQueueARN:
Value: !GetAtt DeadLetterQueue1.Arn
Description: Arn of created SQS
Tried with just 1 queue and its working fine
You have to create the policy separately for the second queue as it will have different !GetAtt DeadLetterQueue1.Arn. If you do not want to do this, you can extract the policy to a different template and use that template as a nested stack in your template with queue.
Related
I have a sqs queue, which has an existing sqs access policy.
I am trying to add another policy statement inside the existing policy, but running my cloudformation template results in overwriting the existing policy with the new policy.
DlqSqsQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub name
KmsMasterKeyId: alias/aws/sqs
KmsDataKeyReusePeriodSeconds: 86400
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
DlqSqsQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev1
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v1
Version: '2012-10-17'
Queues:
- Ref: DlqSqsQueue
DlqSqsQueuePolicyV2:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev2
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v2
Version: '2012-10-17'
Queues:
- Ref: DlqSqsQueue
An SQS queue can have only one AWS::SQS::QueuePolicy. You have to take your Statement from DlqSqsQueuePolicyV2 and add it to DlqSqsQueuePolicy:
DlqSqsQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub name
KmsMasterKeyId: alias/aws/sqs
KmsDataKeyReusePeriodSeconds: 86400
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
DlqSqsQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev1
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v1
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev2
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v2
Version: '2012-10-17'
Queues:
- Ref: DlqSqsQueue
I've recently started learning and implementing services using AWS services. So, I guess I'm missing some small steps which I can't figure it.
I'm trying to implement the following diagram using the Cloudformation template. Everything is working fine unless. The Lambda and SQS queue are subscribed to the SNS topic successfully. Whenever a file is stored at the bucket, or even when I publish a message to the SNS topic manually, the lambda function is triggered successfully, but the message is not published to the SQS queue. I've also added the AWS::SQS::QueuePolicy to allow SNS to send messages to SQS, but it still does not work.
template.yml:
...
Resources:
S3ObjectPutTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${AppName}-vrp-creation-${Environment}-topic
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
...
Bucket:
Type: AWS::S3::Bucket
...
Lambda:
Type: AWS::Serverless::Function
...
Queue:
Type: AWS::SQS::Queue
Properties:
DelaySeconds: 0
MaximumMessageSize: 262144
MessageRetentionPeriod: 864000
QueueName: !Sub ${AppName}-${Environment}-queue
ReceiveMessageWaitTimeSeconds: 0
VisibilityTimeout: 90
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
TopicToQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref S3ObjectPutTopic
Endpoint: !GetAtt Queue.Arn
RawMessageDelivery: true
The full Cloudformation template.yaml file: template.yaml
You have mentioned Service: s3.amazonaws.com instead of Service: sns.amazonaws.com in your SQS policy. Update the template and try.
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
I am trying to create an SQS queue and its associated access policy using cloudformation. Tried a few iterations but it keeps giving me this error:
Value of property Queues must be of type List of String
Below is my template. Can anyone help me point the issue in this:
SQSQueue:
Type: "AWS::SQS::Queue"
Properties:
DelaySeconds: "0"
MaximumMessageSize: "262144"
MessageRetentionPeriod: "10800"
ReceiveMessageWaitTimeSeconds: "0"
VisibilityTimeout: "30"
QueueName: "ScanQueueItems"
DocSQSSNSPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Id: MessageToSQSPolicy
Statement:
Effect: Allow
Principal: "*"
Action:
- SQS:SendMessage
Resource: !GetAtt SQSQueue.Arn
Queues: !Ref SQSQueue
Queues should be List of String. This means that instead of:
Queues: !Ref SQSQueue
you should have:
Queues:
- !Ref SQSQueue
or shorter:
Queues: [!Ref SQSQueue]
Here is example for lambda invocation by event:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
EventPattern:
source:
- "aws.ec2"
detail-type:
- "EC2 Instance State-change Notification"
detail:
state:
- "stopping"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "EventRule"
- "Arn"
How to modify this tp be able to invoke state machine not lambda? I have replaced "LambdaFunction with "MyStateMachine"
but how modify PermissionForEventsToInvokeLambda? There is no AWS::StepFunctions::Permission in cloudformation :-(
Here's an snippet from the template we used to have to trigger state machine processing from an S3 event:
InboundBucketPutObjectRule:
Type: 'AWS::Events::Rule'
Properties:
Name: !Sub 'inbound-bucket-put-object-rule'
EventPattern:
source:
- aws.s3
detail-type:
- AWS API Call via CloudTrail
detail:
eventSource:
- s3.amazonaws.com
eventName:
- PutObject
- CopyObject
- CompleteMultipartUpload
requestParameters:
bucketName:
- !Ref InboundBucket
Targets:
- Id: ProcessNewObject
Arn: !Ref StateMachine
RoleArn: !GetAtt
- StateMachineStartExecutionRole
- Arn
StateMachineStartExecutionPolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
ManagedPolicyName: !Sub 'state-machine-start-execution'
Roles:
- !Ref StateMachineStartExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'states:StartExecution'
Resource:
- !Ref StateMachine
StateMachineStartExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Sub 'state-machine-start-execution'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sts:AssumeRole'
StateMachine:
Type: 'AWS::StepFunctions::StateMachine'
Properties:
StateMachineName: !Sub 'state-machine'
Taken from real template so I had to anonymize this which might have introduced errors.
Template structure:
InboundBucketPutObjectRule is the CloudWatch event which gets triggered on file uploads.
StateMachineStartExecutionPolicy + StateMachineStartExecutionRole essentially allow CloudWatch event to start execution of a state machine.
StateMachine - the state machine which should be started (definition omitted).
I wrote above that we used to have this - but not anymore. Please see the following answer for our current solution:
https://stackoverflow.com/a/57563395/303810
I'm creating AWS CloudFormation template to add a lambda function as a life cycle hook. But the CloudFormation template deployment failed with below message:
The Service-Linked Role for this Auto Scaling group is not yet ready for use.
The CF template is written in YAML and the auto scaling group's part is as follows:
ServerGroup:
Type: 'AWS::AutoScaling::AutoScalingGroup'
DependsOn:
- VpcStack
- NodeManagerExecRole
- NodeManagerSnsTopic
Properties:
VPCZoneIdentifier:
- !GetAtt [VpcStack, Outputs.Subnet2Id]
LaunchConfigurationName: !Ref LaunchConfig2
MinSize: '0'
MaxSize: !Ref NodesPerZone
DesiredCapacity: !Ref NodesPerZone
Cooldown: '300'
HealthCheckType: EC2
HealthCheckGracePeriod: '300'
LoadBalancerNames:
- !Ref ElasticLoadBalancer
LifecycleHookSpecificationList:
- LifecycleTransition: 'autoscaling:EC2_INSTANCE_LAUNCHING'
LifecycleHookName: NodeManager
HeartbeatTimeout: 4800
NotificationTargetARN: !Ref NodeManagerSnsTopic
RoleARN: !GetAtt [NodeManagerExecRole, Arn]
The code snippet of NodeManagerExecRole is like this:
NodeManagerExecRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: NodeManager
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- "sns:Publish"
Resource: "arn:aws:sns:*:*:*"
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: 'arn:aws:logs:*:*:*'
I searched in AWS documentation as well as stackoverfolow and didn't find useful information about this error. It's only mentioned here with on detail informaton.
Is there something I'm missing in the template?
You would need to add a trust policy for autoscaling group to post to SNS.
"Principal": {
"Service": "autoscaling.amazonaws.com"
},