Replace custom AWS IAM policies with managed ones - amazon-web-services

I am trying to replace a bunch of custom AWS IAM policies with managed versions. Currently, the template has:
Resources:
...
AutoScalingRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, AutoScalingRole]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole'
ContainerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: !Join ['', [!Ref ServiceName, ContainerSecurityGroup]]
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref ContainerPort
ToPort: !Ref ContainerPort
SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup
LoadBalancerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription:
!Join ['', [!Ref ServiceName, LoadBalancerSecurityGroup]]
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref LoadBalancerPort
ToPort: !Ref LoadBalancerPort
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
...
CubeJsGroup:
Type: AWS::IAM::Group
Properties:
GroupName: staging-cubejs-group
Policies:
- PolicyName: staging-cubejs-s3-policy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- s3:*
- s3-object-lambda:*
Resource: "*"
- PolicyName: staging-cubejs-athena-policy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- athena:*
Resource: "*"
- Effect: Allow
Action:
- glue:CreateDatabase
- glue:DeleteDatabase
- glue:GetDatabase
- glue:GetDatabases
- glue:UpdateDatabase
- glue:CreateTable
- glue:DeleteTable
- glue:BatchDeleteTable
- glue:UpdateTable
- glue:GetTable
- glue:GetTables
- glue:BatchCreatePartition
- glue:CreatePartition
- glue:DeletePartition
- glue:BatchDeletePartition
- glue:UpdatePartition
- glue:GetPartition
- glue:GetPartitions
- glue:BatchGetPartition
Resource: "*"
- Effect: Allow
Action:
- s3:GetBucketLocation
- s3:GetObject
- s3:ListBucket
- s3:ListBucketMultipartUploads
- s3:ListMultipartUploadParts
- s3:AbortMultipartUpload
- s3:CreateBucket
- s3:PutObject
- s3:PutBucketPublicAccessBloc
Resource: "arn:aws:s3:::aws-athena-query-results-*"
- Effect: Allow
Action:
- cloudwatch:PutMetricAlarm
- cloudwatch:DescribeAlarms
- cloudwatch:DeleteAlarms
Resource: "*"
- Effect: Allow
Action:
- lakeformation:GetDataAccess
Resource: "*"
- Effect: Allow
Action:
- sns:ListTopics
- sns:GetTopicAttributes
Resource: "*"
CubeJsUser:
Type: AWS::IAM::User
Properties:
UserName: 'cubejsUser-staging'
Groups:
- !Ref CubeJsGroup
DependsOn:
- CubeJsGroup
CubeJsUserAccessKey:
Type: AWS::IAM::AccessKey
Properties:
Status: Active
UserName: !Ref CubeJsUser
DependsOn:
- CubeJsUser
Outputs:
Endpoint:
Description: Endpoint
Value: !Join ['', ['https://', !Ref DNSRecord]]
CubeJsUserAccessKey:
Description: "CubeJS user stagin access key id"
Value: !Ref CubeJsUserAccessKey
Export:
Name:
Fn::Sub: "${AWS::StackName}-cubejs-access-id"
CubeJsUserSecretAccessKey:
Description: "CubeJS user access key id"
Value: !GetAtt
- CubeJsUserAccessKey
- SecretAccessKey
Export:
Name:
Fn::Sub: "${AWS::StackName}-cubejs-access-secret"
I like how AutoScalingRole is defined using ManagedPolicyArns etc. To me, it seems like S3FullAccess and such could be used to replace at least some custom permissions. Plus, I would like to maybe drop CubeJsGroup altogether. The template defines all of that CubeJS -related things so that CubeJS had access to Athena and to export keys for CubeJS instance to access AWS. Seems like athena:* could be replaced with AmazonAthenaFullAccess managed policy. Right?
How could I re-write this with as few custom policies as possibly, without breaking anything?

Related

Error while deploying MWAA using Cloudformation

I am getting the following error while deploying Managed Workflow using Apache Airflow using Cloudformation. The error doesnt give out much information and hence difficult to debug
2022-09-16 15:25:42 UTC+0530 MwaaEnvironment CREATE_FAILED Resource
handler returned message: "null (Service: Mwaa, Status Code: 403,
Request ID: de70d88f-383a-43bb-91a7-4ebdee31223e)" (RequestToken:
fe034e51-e5e2-73e9-3616-0bd928d64a1f, HandlerErrorCode: AccessDenied)
It shows access denied but not much information about which resource
Snippet of Cloudformation yaml for the mwaa execution policy
#####################################################################################################################
# CREATE MWAA
#####################################################################################################################
MwaaEnvironment:
Type: AWS::MWAA::Environment
DependsOn: MwaaExecutionPolicy
Properties:
Name: !Sub "${AWS::StackName}-MwaaEnvironment"
SourceBucketArn: !GetAtt EnvironmentBucket.Arn
ExecutionRoleArn: !GetAtt MwaaExecutionRole.Arn
DagS3Path: dags
NetworkConfiguration:
SecurityGroupIds:
- !GetAtt SecurityGroup.GroupId
SubnetIds:
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
WebserverAccessMode: PUBLIC_ONLY
MaxWorkers: !Ref MaxWorkerNodes
LoggingConfiguration:
DagProcessingLogs:
LogLevel: !Ref DagProcessingLogs
Enabled: true
SchedulerLogs:
LogLevel: !Ref SchedulerLogsLevel
Enabled: true
TaskLogs:
LogLevel: !Ref TaskLogsLevel
Enabled: true
WorkerLogs:
LogLevel: !Ref WorkerLogsLevel
Enabled: true
WebserverLogs:
LogLevel: !Ref WebserverLogsLevel
Enabled: true
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref VPC
GroupDescription: !Sub "Security Group for Amazon MWAA Environment ${AWS::StackName}-MwaaEnvironment"
GroupName: !Sub "airflow-security-group-${AWS::StackName}-MwaaEnvironment"
SecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !Ref SecurityGroup
IpProtocol: "-1"
SourceSecurityGroupId: !Ref SecurityGroup
SecurityGroupEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !Ref SecurityGroup
IpProtocol: "-1"
CidrIp: "0.0.0.0/0"
MwaaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- airflow-env.amazonaws.com
- airflow.amazonaws.com
Action:
- "sts:AssumeRole"
Path: "/service-role/"
MwaaExecutionPolicy:
DependsOn: EnvironmentBucket
Type: AWS::IAM::ManagedPolicy
Properties:
Roles:
- !Ref MwaaExecutionRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: airflow:PublishMetrics
Resource:
- !Sub "arn:aws:airflow:${AWS::Region}:${AWS::AccountId}:environment/${EnvironmentName}"
- Effect: Deny
Action: s3:ListAllMyBuckets
Resource:
- !Sub "${EnvironmentBucket.Arn}"
- !Sub "${EnvironmentBucket.Arn}/*"
- Effect: Allow
Action:
- "s3:GetObject*"
- "s3:GetBucket*"
- "s3:List*"
Resource:
- !Sub "${EnvironmentBucket.Arn}"
- !Sub "${EnvironmentBucket.Arn}/*"
- Effect: Allow
Action:
- logs:DescribeLogGroups
Resource: "*"
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:CreateLogGroup
- logs:PutLogEvents
- logs:GetLogEvents
- logs:GetLogRecord
- logs:GetLogGroupFields
- logs:GetQueryResults
- logs:DescribeLogGroups
Resource:
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:airflow-${AWS::StackName}*"
- Effect: Allow
Action: cloudwatch:PutMetricData
Resource: "*"
- Effect: Allow
Action:
- sqs:ChangeMessageVisibility
- sqs:DeleteMessage
- sqs:GetQueueAttributes
- sqs:GetQueueUrl
- sqs:ReceiveMessage
- sqs:SendMessage
Resource:
- !Sub "arn:aws:sqs:${AWS::Region}:*:airflow-celery-*"
- Effect: Allow
Action:
- kms:Decrypt
- kms:DescribeKey
- "kms:GenerateDataKey*"
- kms:Encrypt
NotResource: !Sub "arn:aws:kms:*:${AWS::AccountId}:key/*"
Condition:
StringLike:
"kms:ViaService":
- !Sub "sqs.${AWS::Region}.amazonaws.com"
- Action:
- cloudwatch:*
- cloudformation:CreateStack
- cloudformation:DescribeStackEvents
- ec2:AuthorizeSecurityGroupIngress
- ec2:AuthorizeSecurityGroupEgress
- ec2:CancelSpotInstanceRequests
- ec2:CreateRoute
- ec2:CreateSecurityGroup
- ec2:CreateTags
- ec2:DeleteRoute
- ec2:DeleteTags
- ec2:DeleteSecurityGroup
- ec2:DescribeAvailabilityZones
- ec2:DescribeAccountAttributes
- ec2:DescribeInstances
- ec2:DescribeKeyPairs
- ec2:DescribeRouteTables
- ec2:DescribeSecurityGroups
- ec2:DescribeSpotInstanceRequests
- ec2:DescribeSpotPriceHistory
- ec2:DescribeSubnets
- ec2:DescribeVpcAttribute
- ec2:DescribeVpcs
- ec2:DescribeRouteTables
- ec2:DescribeNetworkAcls
- ec2:CreateVpcEndpoint
- ec2:ModifyImageAttribute
- ec2:ModifyInstanceAttribute
- ec2:RequestSpotInstances
- ec2:RevokeSecurityGroupEgress
- ec2:RunInstances
- ec2:TerminateInstances
- elasticmapreduce:*
- iam:GetPolicy
- iam:GetPolicyVersion
- iam:ListRoles
- iam:PassRole
- kms:List*
- s3:*
- sdb:*
Effect: Allow
Resource: "*"
- Effect: Allow
Action: iam:CreateServiceLinkedRole
Resource: "*"
Condition:
StringLike:
iam:AWSServiceName:
- elasticmapreduce.amazonaws.com
- elasticmapreduce.amazonaws.com.cn

Cloudformation !Ref doesn't return the correct value for vpc endpoint

I have a vpc endpoint created with:
ExecuteApiVPCEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal: "*"
Action:
- "execute-api:*"
Resource:
- "*"
ServiceName: !Sub com.amazonaws.${AWS::Region}.execute-api
VpcId: !Ref MyVPC
VpcEndpointType: Interface
SubnetIds:
- !Ref privateSubnet01
- !Ref privateSubnet02
SecurityGroupIds:
- !Ref EndpointSG # has to allow traffic from your VPC
PrivateDnsEnabled: true
and a private API created with:
APIDefinition:
Type: 'AWS::Serverless::Api'
Properties:
StageName: {Ref: Stage}
EndpointConfiguration: PRIVATE
Auth:
ResourcePolicy:
CustomStatements:
- Action: ['execute-api:Invoke']
Effect: Allow
Principal: '*'
Resource: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:/*/*/*"
- Action: ['execute-api:Invoke']
Effect: Deny
Principal: '*'
Resource: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:/*/*/*"
Condition:
ForAllValues:StringNotEquals:
aws:sourceVpce:
- !Ref ExecuteApiVPCEndpoint
The stack was deployed with no error, but the vpce in the resource policy of the API doesn't match with the vpce being created. What's going wrong here? Thanks!

Attach IAM role using cloudformation

I am trying to create a webserver (Ec2 instance) with an IAM role of access to S3 bucket using ccloudformation. It creates the role but doesn't attach the role to ec2-instance.
webserver:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: sg-webserver
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
Description: For traffic from Internet
GroupDescription: Security Group for demo server
VpcId: !Ref VPC
EC2Instance:
Type: AWS::EC2::Instance
Properties:
AvailabilityZone: us-east-2a
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
DeleteOnTermination: "true"
VolumeSize: "8"
VolumeType: gp2
ImageId: ami-0bdcc6c05dec346bf
InstanceType: t2.micro
KeyName: key-webserver
NetworkInterfaces:
- Description: Primary network interface
DeviceIndex: 0
SubnetId: !Ref SubnetA
GroupSet:
- Ref: webserver
ListS3BucketsInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles:
- Ref: S3FullAccess
ListS3BucketsPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: ListS3BucketsPolicy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- s3:List*
Resource: "*"
Roles:
- Ref: S3FullAccess
S3FullAccess:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
The yml file works fine. It creates an ec2-instance and security group but doesnot attach the IAM role to the ec2 Instance
You're not attaching it to the EC2 resource, it should instead add it via the IamInstanceProfile property of EC2.
The value should be !Ref ListS3BucketsInstanceProfile
The instance profile will never automatically attach to the instance.
webserver:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: sg-webserver
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
Description: For traffic from Internet
GroupDescription: Security Group for demo server
VpcId: !Ref VPC
EC2Instance:
Type: AWS::EC2::Instance
Properties:
AvailabilityZone: us-east-2a
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
DeleteOnTermination: "true"
VolumeSize: "8"
VolumeType: gp2
ImageId: ami-0bdcc6c05dec346bf
InstanceType: t2.micro
IamInstanceProfile: !Ref ListS3BucketsInstanceProfile
KeyName: key-webserver
NetworkInterfaces:
- Description: Primary network interface
DeviceIndex: 0
SubnetId: !Ref SubnetA
GroupSet:
- Ref: webserver
ListS3BucketsInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles:
- Ref: S3FullAccess
ListS3BucketsPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: ListS3BucketsPolicy
PolicyDocument:
Statement:
- Effect: Allow
Action:
- s3:List*
Resource: "*"
Roles:
- Ref: S3FullAccess
S3FullAccess:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
For more information over instance profiles take a look at this link.
The missing piece is the IamInstanceProfile parameter to EC2Instance. That should reference the instance profile ARN. Add the following to EC2Instance:
IamInstanceProfile: !GetAtt ListS3BucketsInstanceProfile.Arn

error while creating a stack using cloudformation to launch emr through lambda

I am trying to create a stack using cloudformation to launch an emr by triggering a lambda function. But when the stack is getting created I am having this error for creating ManagedApiPolicy:
Resource a36ef9be-97e1-4949-9b04-c1979eda5955 must be in ARN format or "*"
Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument;
Where am I going wrong?
My code is:
AWSTemplateFormatVersion: 2010-09-09
Description: EMR Cluster for Spark
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Common Parameters
Parameters:
- SystemValue
- SubSystemValue
- RevenueStreamValue
- EnvironmentValue
- KMSArn
- Label:
default: EMR Parameters
default: Lambda Parameters
Parameters:
- EMRClusterName
- KeyName
- MasterInstanceType
- CoreInstanceType
- NumberOfCoreInstances
- VpcId
- VPCSubnetIds
- LogUriFolder
- S3DataUri
- ReleaseLabel
- ApplicationsList
- PackageBucketName
- EbsRootVolumeSize
- FunctionName1Value
- Label:
default: S3 Parameters
Parameters:
- EDWBucketName
ParameterLabels:
SystemValue:
default: System
SubSystemValue:
default: SubSystem
RevenueStreamValue:
default: Revenue Stream
EnvironmentValue:
default: Environment Value
FunctionName1Value:
default: Lambda-1 Name
EMRClusterName:
default: EMR Cluster Name
KeyName:
default: Key Name
MasterInstanceType:
default: Master Instance Type
CoreInstanceType:
default: Core Instance Type
NumberOfCoreInstances:
default: Number Of Core Instances
VpcId:
default: VPC ID
VPCSubnetIds:
default: VPC Subnet ID
LogUriFolder:
default: Log Uri Folder
S3DataUri:
default: S3 Data Uri
ReleaseLabel:
default: Release Label
ApplicationsList:
default: Applications List
KMSArn:
default: KMS Arn
EDWBucketName:
default: EDW Bucket Name
PackageBucketName:
default: Package Bucket Name
EbsRootVolumeSize:
default: Ebs Root Volume Size
#########################################################################
Parameters:
KMSArn:
Type: String
Description: Enter KMS ARN based on your subsystem.
Default: 'a36ef9be-97e1-4949-9b04-c1979eda5955'
SystemValue:
Type: String
Description: Enter System Name
Default: 'Messaging'
SubSystemValue:
Type: String
Description: Enter SubSystem Name
Default: 'edw'
RevenueStreamValue:
Type: String
Description: Enter Revenue Stream Name
Default: 'edw'
FunctionName1Value:
Type: String
Default: 'EMRCluster'
Description: Enter 1st Lambda Function Name(Do not append System & Sub-System Name).
EDWBucketName:
Type: String
Default: 'crx-dev-messaging-edw'
Description: Enter crx-[env]-messaging-edw
PackageBucketName:
Type: String
Default: 'crx-dev-deployments'
Description: Enter crx-[env]-deployments
EnvironmentValue:
Type: String
Default: 'dev'
LambdaRuntime:
Type: String
Default: 'Python-2.7'
TagEnvironmentValue:
Type: String
Default: 'dev'
VpcId:
Type: 'AWS::EC2::VPC::Id'
Default: vpc-7c368507
VpcCIDR:
Type: String
Default: '10.10.16.0/20'
VPCSubnetIds:
Type: AWS::EC2::Subnet::Id
Default: subnet-4d527c62
Env:
Type: String
Default: 'dev'
KeyName:
Type: AWS::EC2::KeyPair::KeyName
Description: Ane<Env>-Messaging-EDW
Default: development
MasterInstanceType:
Type: String
Description: Instance type to be used for the master instance.
Default: 'm5.xlarge'
AllowedValues:
- m5.xlarge
- m1.medium
CoreInstanceType:
Type: String
Description: Instance type to be used for core instances.
Default: 'm5.xlarge'
AllowedValues:
- m5.xlarge
- m1.medium
NumberOfCoreInstances:
Description: Must be a valid number - 2
Type: Number
Default: '2'
LogUriFolder:
Type: String
Description: S3 Folder name for spark logs (spark-logs)
Default: 's3n://crx-dev-messaging-edw/spark-logs/'
S3DataUri:
Type: String
Description: Must be a valid S3 bucket URL
Default: 's3n://crx-messaging-edw'
EbsRootVolumeSize:
Type: String
Description: Specify the volume size
Default: '200'
ReleaseLabel:
Type: String
Description: Must be a valid EMR release version emr-6.0.0
Default: 'emr-6.0.0'
ApplicationsList:
Type: String
Description: Please select which application will be installed on the cluster
this would be either Ganglia,spark,Hadoop etc.,
Default: 'Spark'
AllowedValues:
- Hadoop
- Ganglia
- Spark
#########################################################################
Resources:
BlockPublicAclsucket1:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Sub 'crx-test-${EnvironmentValue}-${SubSystemValue}'
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: !Ref KMSArn
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
#EbsRootVolumeSize: !Ref EbsRootVolumeSize
SvcAccessSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: !Sub SG-${Env}-${SystemValue}-${SubSystemValue}-ServiceAccess
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-Service Access Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-ServiceAccess'
EmrMasterSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRMaster'
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-EMR Managed Master Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRMaster'
EmrSlaveSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRSlave'
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-EMR Managed Slave Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRSlave'
LambdaSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}'
VpcId: !Ref VpcId
GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-Lambda Security Group'
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: !Ref VpcCIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
Tags:
- Key: Environment
Value: !Ref TagEnvironmentValue
- Key: RevenueStream
Value: !Ref RevenueStreamValue
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Key: Name
Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}'
Lambda1:
Type: 'AWS::Lambda::Function'
DependsOn:
- LambdaExecutionRole
- ManagerApiPolicy
Properties:
Description: Script to launch EMR
FunctionName: !Sub ' ${SystemValue}-${SubSystemValue}-${FunctionName1Value}'
Handler: lambda_function.lambda_handler
Code:
S3Bucket: !Ref PackageBucketName
S3Key: emr-launch.zip
Role: !GetAtt
- LambdaExecutionRole
- Arn
Runtime: !Ref LambdaRuntime
Timeout: '900'
MemorySize: 512
Environment:
Variables:
parameterPath: !Sub '/crx/${SystemValue}/${SubSystemValue}/'
region: !Ref 'AWS::Region'
VpcConfig:
SubnetIds: !Ref VPCSubnetIds
SecurityGroupIds:
- !GetAtt
- LambdaSecurityGroup
- GroupId
Tags:
- Value: !Sub '${SystemValue}-${SubSystemValue}-${FunctionName1Value}'
Key: Name
- Key: SubSystem
Value: !Ref SubSystemValue
- Key: System
Value: !Ref SystemValue
- Value: !Ref RevenueStreamValue
Key: RevenueStream
- Value: !Ref TagEnvironmentValue
Key: Environment
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
DependsOn:
- LambdaSecurityGroup
Properties:
RoleName: !Sub '${SystemValue}-${SubSystemValue}'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- states.amazonaws.com
- events.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagerApiPolicy:
Type: 'AWS::IAM::ManagedPolicy'
DependsOn:
- LambdaExecutionRole
Properties:
ManagedPolicyName: !Sub '${SystemValue}-${SubSystemValue}'
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: common
Effect: Allow
Action:
- 'ec2:CreateNetworkInterface'
- 'ec2:Describe*'
- 'ec2:Get*'
- 'ec2:DeleteNetworkInterface'
- 'kms:GenerateDataKey'
- 'kms:ListAliases'
- 'kms:ListKeys'
- 'kms:ReEncryptTo'
- 'kms:CreateKey'
- 'kms:GenerateRandom'
- 'kms:ReEncryptFrom'
- 'logs:CreateLogGroup'
- 'logs:PutLogEvents'
- 'logs:CreateLogStream'
- 'ssm:DescribeParameters'
- 'lambda:InvokeFunction'
Resource: '*'
- Sid: ssm
Effect: Allow
Action:
- 'ssm:GetParametersByPath'
- 'ssm:GetParameters'
- 'ssm:GetParameter'
Resource:
- !Join
- ''
- - 'arn:aws:ssm:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':parameter/crx/'
- !Sub '${SystemValue}/${SubSystemValue}*'
- Sid: kms
Effect: Allow
Action: 'kms:*'
Resource:
- !Ref KMSArn
- !Join
- ''
- - 'arn:aws:kms:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':alias/'
- !Sub '${SystemValue}-${SubSystemValue}'
- Sid: s3
Effect: Allow
Action:
- 's3:PutObject'
- 's3:GetObject'
- 's3:List*'
Resource:
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-${PackageBucketName}'
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-${PackageBucketName}/*'
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-public-assets'
- !Join
- ''
- - 'arn:aws:s3:'
- '::'
- !Sub 'crx-${EnvironmentValue}-public-assets/*'
Roles:
- !Ref LambdaExecutionRole
# LambdaExecutionRoleforEMRPolicy:
# DependsOn:
# - LambdaExecutionRole
# Type: 'AWS::IAM::Policy'
# Properties:
# PolicyName: LambdaforEMRRolePolicy
# Roles:
# !Ref LambdaExecutionRole
# AssumeRolePolicyDocument:
# Version: 2012-10-17
# Statement:
# Action:
# - 'elasticmapreduce: *'
# Resource: '*'
# - Effect: Allow
# Action:
# - logs:CreateLogGroup
# - logs:CreateLogStream
# - logs:PutLogEvents
# - Resource: arn:aws:*:*:*
# - Effect: Allow
# Action:
# - elasticmapreduce:'*'
# - Resource: '*'
PermissionForEventsToInvokeLambda:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName:
!Ref LambdaExecutionRole
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
SourceArn: !GetAtt
- Lambda1
- Arn
EMRClusterServiceRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role
Path: /
EMRClusterinstanceProfileRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role
Path: /
EMRClusterinstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref EMRClusterinstanceProfileRole
I think in
- Sid: kms
Effect: Allow
Action: 'kms:*'
Resource:
- !Ref KMSArn
KMSArn defaults to the resource id and not to the resource arn:
Parameters:
KMSArn:
Type: String
Description: Enter KMS ARN based on your subsystem.
Default: 'a36ef9be-97e1-4949-9b04-c1979eda5955'
So you can build an arn of the resource id using join.
I think in the other usage of KMSArn in
Resources:
BlockPublicAclsucket1:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: !Sub 'crx-test-${EnvironmentValue}-${SubSystemValue}'
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: !Ref KMSArn
you can also provide an arn according to the documentation. Alternatively you could change the default of KMSArn from resource id to arn.

CloudFormation Response Object Timeout

I've got a CloudFormation Lambda Backed Custom Resource ,
Lambda function in public subnets but when I check the cloudWatch logs shown it below
Log-Message#1
Starting new HTTPS connection (1): cloudformation-custom-resource-response-eucentral1.s3.eu-central-1.amazonaws.com
Log-Message#2
Task timed out after 30.03 seconds
How I can handle this problem , my cloudformation is shown as below .
Resources:
HelloWorld: #Custom Resource
Type: Custom::HelloWorld
Properties:
ServiceToken:
Fn::GetAtt:
- TestFunction #Reference to Function to be run
- Arn #ARN of the function to be run
Input1:
Ref: Message
TestFunction: #Lambda Function
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket:
Ref: S3Bucket
S3Key:
Ref: S3Key
Handler:
Fn::Join:
- ''
- - Ref: ModuleName
- ".lambda_handler"
Role:
Fn::GetAtt:
- LambdaExecutionRole
- Arn
VpcConfig:
SecurityGroupIds:
- !Ref SecurityGroup
SubnetIds:
- Fn::Select: [ 0, !Ref PublicSubnet1 ]
- Fn::Select: [ 0, !Ref PublicSubnet2 ]
Runtime: python2.7
Timeout: '30'
LambdaExecutionRole: #IAM Role for Custom Resource
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
Resource: "*"
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: "sec_group_name"
GroupDescription: "SSH traffic in, all traffic out."
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
My subnets routing table associated with InternetGateway, but it giving CloudFormationResponse object error , How I can solve this connection problem .
Help ! Thanks :))
I am guessing your public subnet does not have a NAT gateway or NAT instance attached to it (InternetGateway alone is not enogh). As per AWS, this is required. If your functions does not need general internet access but access to AWS resources, you should consider VPC Endpoints. They are cheaper, but not sure if available for all resources.