I created a serverless resource for elasticsearch with all necessary policies. We are using seed to build and deploy to cloudformation stacks. The policy is created, elasticsearch domain creation is in process and the policy is also attached to the domain. But it takes very long time and even waiting an hour or so, the status is still “CREATE_IN_PROGRESS”. The process doesn't complete.
I have tried without policies, which creates the elasticsearch domain without taking so much time.
Any suggestions?
Update: Serverless Resource
Resources:
ElasticSearchInstance:
Type: AWS::Elasticsearch::Domain
Properties:
ElasticsearchVersion: 7.4
DomainName: ${self:custom.stage}-domainName
EBSOptions:
EBSEnabled: true
VolumeType: gp2
VolumeSize: 10
ElasticsearchClusterConfig:
InstanceType: "t2.small.elasticsearch"
InstanceCount: 1
DedicatedMasterEnabled: false
ZoneAwarenessEnabled: false
CognitoOptions:
Enabled: true
IdentityPoolId:
Ref: CognitoIdentityPool
RoleArn:
Fn::GetAtt: [ElasticSearchLambdaServiceRole, Arn]
UserPoolId:
Ref: CognitoUserPool
ElasticSearchLambdaServiceRole:
Type: AWS::IAM::Role
Properties:
RoleName: ElasticSearchLambdaServiceRole
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- es.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: ElasticSearchAccessPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
# # Policy same as "AmazonESCognitoAccess"
- Effect: "Allow"
Action:
- "cognito-idp:DescribeUserPool"
- "cognito-idp:DescribeUserPoolClient"
- "cognito-idp:AdminInitiateAuth"
- "cognito-idp:AdminUserGlobalSignOut"
- "cognito-idp:ListUserPoolClients"
- "cognito-identity:DescribeIdentityPool"
- "cognito-identity:SetIdentityPoolRoles"
- "cognito-identity:GetIdentityPoolRoles"
Resource: "*"
- Effect: "Allow"
Action: "iam:PassRole"
Resource: "*"
Condition:
StringLike:
"iam:PassedToService": "cognito-identity.amazonaws.com"
- Effect: Allow
Action: "es:ESHttpGet"
Resource: "*"
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- 'Fn::Join':
- ':'
-
- 'arn:aws:logs'
- Ref: 'AWS::Region'
- Ref: 'AWS::AccountId'
- 'log-group~:/aws/lambda/*:*:*'
- Effect: "Allow"
Action:
- "es:*"
Resource:
- 'Fn::Join':
- ''
-
- 'arn:aws:es:'
- Ref: 'AWS::Region'
- ':'
- Ref: 'AWS::AccountId'
- ':domain/'
- "${self:custom.stage}-domainName"
- '/*'
Related
I have a sqs queue, which has an existing sqs access policy.
I am trying to add another policy statement inside the existing policy, but running my cloudformation template results in overwriting the existing policy with the new policy.
DlqSqsQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub name
KmsMasterKeyId: alias/aws/sqs
KmsDataKeyReusePeriodSeconds: 86400
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
DlqSqsQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev1
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v1
Version: '2012-10-17'
Queues:
- Ref: DlqSqsQueue
DlqSqsQueuePolicyV2:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev2
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v2
Version: '2012-10-17'
Queues:
- Ref: DlqSqsQueue
An SQS queue can have only one AWS::SQS::QueuePolicy. You have to take your Statement from DlqSqsQueuePolicyV2 and add it to DlqSqsQueuePolicy:
DlqSqsQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub name
KmsMasterKeyId: alias/aws/sqs
KmsDataKeyReusePeriodSeconds: 86400
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
DlqSqsQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev1
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v1
- Action: sqs:SendMessage
Condition:
ArnEquals:
aws:SourceArn:
Fn::GetAtt:
- rulenamev2
- Arn
Effect: Allow
Principal:
Service: events.amazonaws.com
Resource:
Fn::GetAtt:
- DlqSqsQueue
- Arn
Sid: v2
Version: '2012-10-17'
Queues:
- Ref: DlqSqsQueue
I am getting the following error while deploying Managed Workflow using Apache Airflow using Cloudformation. The error doesnt give out much information and hence difficult to debug
2022-09-16 15:25:42 UTC+0530 MwaaEnvironment CREATE_FAILED Resource
handler returned message: "null (Service: Mwaa, Status Code: 403,
Request ID: de70d88f-383a-43bb-91a7-4ebdee31223e)" (RequestToken:
fe034e51-e5e2-73e9-3616-0bd928d64a1f, HandlerErrorCode: AccessDenied)
It shows access denied but not much information about which resource
Snippet of Cloudformation yaml for the mwaa execution policy
#####################################################################################################################
# CREATE MWAA
#####################################################################################################################
MwaaEnvironment:
Type: AWS::MWAA::Environment
DependsOn: MwaaExecutionPolicy
Properties:
Name: !Sub "${AWS::StackName}-MwaaEnvironment"
SourceBucketArn: !GetAtt EnvironmentBucket.Arn
ExecutionRoleArn: !GetAtt MwaaExecutionRole.Arn
DagS3Path: dags
NetworkConfiguration:
SecurityGroupIds:
- !GetAtt SecurityGroup.GroupId
SubnetIds:
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
WebserverAccessMode: PUBLIC_ONLY
MaxWorkers: !Ref MaxWorkerNodes
LoggingConfiguration:
DagProcessingLogs:
LogLevel: !Ref DagProcessingLogs
Enabled: true
SchedulerLogs:
LogLevel: !Ref SchedulerLogsLevel
Enabled: true
TaskLogs:
LogLevel: !Ref TaskLogsLevel
Enabled: true
WorkerLogs:
LogLevel: !Ref WorkerLogsLevel
Enabled: true
WebserverLogs:
LogLevel: !Ref WebserverLogsLevel
Enabled: true
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VPC
GroupDescription: !Sub "Security Group for Amazon MWAA Environment ${AWS::StackName}-MwaaEnvironment"
GroupName: !Sub "airflow-security-group-${AWS::StackName}-MwaaEnvironment"
SecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref SecurityGroup
IpProtocol: "-1"
SourceSecurityGroupId: !Ref SecurityGroup
SecurityGroupEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !Ref SecurityGroup
IpProtocol: "-1"
CidrIp: "0.0.0.0/0"
MwaaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- airflow-env.amazonaws.com
- airflow.amazonaws.com
Action:
- "sts:AssumeRole"
Path: "/service-role/"
MwaaExecutionPolicy:
DependsOn: EnvironmentBucket
Type: AWS::IAM::ManagedPolicy
Properties:
Roles:
- !Ref MwaaExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: airflow:PublishMetrics
Resource:
- !Sub "arn:aws:airflow:${AWS::Region}:${AWS::AccountId}:environment/${EnvironmentName}"
- Effect: Deny
Action: s3:ListAllMyBuckets
Resource:
- !Sub "${EnvironmentBucket.Arn}"
- !Sub "${EnvironmentBucket.Arn}/*"
- Effect: Allow
Action:
- "s3:GetObject*"
- "s3:GetBucket*"
- "s3:List*"
Resource:
- !Sub "${EnvironmentBucket.Arn}"
- !Sub "${EnvironmentBucket.Arn}/*"
- Effect: Allow
Action:
- logs:DescribeLogGroups
Resource: "*"
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:CreateLogGroup
- logs:PutLogEvents
- logs:GetLogEvents
- logs:GetLogRecord
- logs:GetLogGroupFields
- logs:GetQueryResults
- logs:DescribeLogGroups
Resource:
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:airflow-${AWS::StackName}*"
- Effect: Allow
Action: cloudwatch:PutMetricData
Resource: "*"
- Effect: Allow
Action:
- sqs:ChangeMessageVisibility
- sqs:DeleteMessage
- sqs:GetQueueAttributes
- sqs:GetQueueUrl
- sqs:ReceiveMessage
- sqs:SendMessage
Resource:
- !Sub "arn:aws:sqs:${AWS::Region}:*:airflow-celery-*"
- Effect: Allow
Action:
- kms:Decrypt
- kms:DescribeKey
- "kms:GenerateDataKey*"
- kms:Encrypt
NotResource: !Sub "arn:aws:kms:*:${AWS::AccountId}:key/*"
Condition:
StringLike:
"kms:ViaService":
- !Sub "sqs.${AWS::Region}.amazonaws.com"
- Action:
- cloudwatch:*
- cloudformation:CreateStack
- cloudformation:DescribeStackEvents
- ec2:AuthorizeSecurityGroupIngress
- ec2:AuthorizeSecurityGroupEgress
- ec2:CancelSpotInstanceRequests
- ec2:CreateRoute
- ec2:CreateSecurityGroup
- ec2:CreateTags
- ec2:DeleteRoute
- ec2:DeleteTags
- ec2:DeleteSecurityGroup
- ec2:DescribeAvailabilityZones
- ec2:DescribeAccountAttributes
- ec2:DescribeInstances
- ec2:DescribeKeyPairs
- ec2:DescribeRouteTables
- ec2:DescribeSecurityGroups
- ec2:DescribeSpotInstanceRequests
- ec2:DescribeSpotPriceHistory
- ec2:DescribeSubnets
- ec2:DescribeVpcAttribute
- ec2:DescribeVpcs
- ec2:DescribeRouteTables
- ec2:DescribeNetworkAcls
- ec2:CreateVpcEndpoint
- ec2:ModifyImageAttribute
- ec2:ModifyInstanceAttribute
- ec2:RequestSpotInstances
- ec2:RevokeSecurityGroupEgress
- ec2:RunInstances
- ec2:TerminateInstances
- elasticmapreduce:*
- iam:GetPolicy
- iam:GetPolicyVersion
- iam:ListRoles
- iam:PassRole
- kms:List*
- s3:*
- sdb:*
Effect: Allow
Resource: "*"
- Effect: Allow
Action: iam:CreateServiceLinkedRole
Resource: "*"
Condition:
StringLike:
iam:AWSServiceName:
- elasticmapreduce.amazonaws.com
- elasticmapreduce.amazonaws.com.cn
I am creating some IAM roles, policies via cloudformation but I would like to add policies based on the condition I have, say if it is dev then i would like to add certain policy statement. any suggestions ?
Parameters:
environment:
Type: String
Default: dev
AllowedValues:
- dev
- prd
Condition:
isDev: !Equals [ !Ref environment, dev]
Resources:
StandAlonePolicy:
Type: AWS::IAM::Policy
Properties:
#How to add a condition - isDev
PolicyName: "s3-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Resource: "*"
Action:
- "s3:Get*"
You can do this using If:
Parameters:
environment:
Type: String
Default: dev
AllowedValues:
- dev
- prd
Conditions:
isDev: !Equals [ !Ref environment, dev]
Resources:
StandAlonePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: "s3-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Resource: "*"
Action:
- "s3:Get*"
- !If
- isDev
- Sid: new-statement-for-dev-only
Effect: Allow
Resource: "*"
Action:
- "s3:Put*"
- !Ref "AWS::NoValue"
I am trying to create a KMS Key using Cloudformation unfortunately I am not able to create it. In the console I am getting the following error :
null (Service: Kms, Status Code: 400, Request ID: 156b452d-8ffb-5517-9jbc-a6yh6e3a79, Extended Request ID: null)
I am not able to understand the root cause of the issue. Please refer to the attached template which I am using to create the KMS :
AWSTemplateFormatVersion: 2010-09-09
Description: Testing KMS Using CloudFormation
Resources:
KMSEncryption:
Type: AWS::KMS::Key
Properties:
Description: KMS-Key
KeyPolicy:
Version: '2012-10-17'
Id: encryption-key
EnableKeyRotation: 'True'
PendingWindowInDays: 7
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS:
Fn::Join:
- ''
- - 'arn:aws:iam::'
- Ref: AWS::AccountId
- :root
Action: kms:*
Resource: '*'
- Sid: Allow use of the key
Effect: Allow
Principal:
AWS:
Fn::Join:
- ''
- - 'arn:aws:iam::'
- Ref: AWS::AccountId
- :role/
- !Ref KMSLambdaRole
Action:
- kms:DescribeKey
- kms:Encrypt
- kms:Decrypt
- kms:ReEncrypt*
- kms:GenerateDataKey
- kms:GenerateDataKeyWithoutPlaintext
Resource: '*'
- Sid: Allow administration of the key
Effect: Allow
Principal:
AWS: arn:aws:iam::xxxxxxxxx:user/Shiv
Action:
- kms:Create*
- kms:Describe*
- kms:Enable*
- kms:List*
- kms:Put*
- kms:Update*
- kms:Revoke*
- kms:Disable*
- kms:Get*
- kms:Delete*
- kms:ScheduleKeyDeletion
- kms:CancelKeyDeletion
EncryptionAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: 'Testing'
TargetKeyId:
Ref: KMSEncryption
KMSLambdaRole:
Type: AWS::IAM::Role
Properties:
RoleName: 'TestingKMSAccess'
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/ReadOnlyAccess
Policies:
- PolicyName: AWSLambdaBasicExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: SQS
Action:
- 'sqs:SendMessage'
- 'sqs:SendMessageBatch'
Effect: Allow
Resource: '*'
Your EnableKeyRotation and PendingWindowInDays should be outside of KeyPolicy:
Resources:
KMSEncryption:
Type: AWS::KMS::Key
Properties:
Description: KMS-Key
EnableKeyRotation: 'True'
PendingWindowInDays: 7
KeyPolicy:
Version: '2012-10-17'
Id: encryption-key
# the rest
Note, that there could be other issues which are not yet apparent, e.g. non-existing principles.
I have 2 policies each for S3 and Kinesis stream which includes DescribeStream. The S3 policy works well but I am getting this error with KinesisPolicy.
Resources:
S3
KinesisStream
Firehose
Role:
FirehoseRole
Policies:
S3 policy with the following permissions:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Kinesis Policy with the following permissions:
- 'kinesis:PutRecord'
- 'kinesis:DescribeStreamSummary'
- 'kinesis:PutRecords'
- 'kinesis:GetShardIterator'
- 'kinesis:GetRecords'
- 'kinesis:DescribeStream'
Error:
The role (firehoseRole) is not authorized to perform DescribeStream on MyKinesisStream.
Cloud formation template
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
VersioningConfiguration:
Status: Enabled
firehoseRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: 'sts:AssumeRole'
Condition:
StringEquals:
'sts:ExternalId': !Ref 'AWS::AccountId'
DeliveryPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: firehose_delivery_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Resource:
- !Sub 'arn:aws:s3:::${S3Bucket}'
- !Sub 'arn:aws:s3:::${S3Bucket}*'
Roles:
- !Ref firehoseRole
KinesisPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: kinesis_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'kinesis:PutRecord'
- 'kinesis:DescribeStreamSummary'
- 'kinesis:PutRecords'
- 'kinesis:GetShardIterator'
- 'kinesis:GetRecords'
- 'kinesis:DescribeStream'
Resource:
- !GetAtt MyKinesisStream.Arn
Roles:
- !Ref firehoseRole
MyKinesisStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 1
DeliveryStream:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
DeliveryStreamType: KinesisStreamAsSource
KinesisStreamSourceConfiguration:
KinesisStreamARN: !GetAtt MyKinesisStream.Arn
RoleARN: !GetAtt firehoseRole.Arn
S3DestinationConfiguration:
BucketARN: !GetAtt S3Bucket.Arn
BufferingHints:
IntervalInSeconds: 60
SizeInMBs: 50
CompressionFormat: UNCOMPRESSED
Prefix: firehose/
RoleARN: !GetAtt firehoseRole.Arn
I was able to resolve the error. I had to add DependsOn To DeliveryStream and include both the policies.