I have the below config to read data from S3 written there by Kinesis Firehose:
S3AthenaStore:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.s3AthenaStore}
AnalysisGlueDatabase:
Type: AWS::Glue::Database
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseInput:
Name: !Join
- ''
- - '${self:custom.glueName}-'
- 'db'
Description: "Analysis aws Glue database"
DependsOn:
- S3AthenaStore
AnalyticsGlueRole:
Type: AWS::IAM::Role
DependsOn:
- S3AnalyticsStore
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "glue.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
ManagedPolicyArns:
['arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole']
Policies:
-
PolicyName: "S3BucketAccessPolicy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "s3:GetObject"
- "s3:PutObject"
Resource:
- !Join
- ''
- - !GetAtt S3AnalyticsStore.Arn
- "*"
AnalyticsGlueCrawler:
Type: AWS::Glue::Crawler
Properties:
Name: "AnalysisCrawler"
Role: !GetAtt AnalyticsGlueRole.Arn
DatabaseName: !Ref AnalysisGlueDatabase
Targets:
S3Targets:
- Path: !Ref S3AnalyticsStore
SchemaChangePolicy:
UpdateBehavior: "LOG"
DeleteBehavior: "LOG"
Schedule:
ScheduleExpression: "cron(00 0/1 * * ? *)"
RecrawlPolicy:
RecrawlBehavior: CRAWL_NEW_FOLDERS_ONLY
DependsOn:
- AnalyticsGlueRole
- AnalysisGlueDatabase
AnalyticsAthenaWorkGroup:
Type: AWS::Athena::WorkGroup
Properties:
Name: ${self:service}-${self:provider.stage}-wg
WorkGroupConfiguration:
ResultConfiguration:
OutputLocation:
!Join
- ''
- - 's3://'
- !Ref S3AthenaStore
DependsOn:
- S3AthenaStore
The data is the folders with the following pattern: ${bucket}/${year}/${month}/${date}/${hour}/event-collection-stream-staging-deliver-1-2022-07-14-23-51-22-cdb2f06a-e825-47d0-a781-efd4195ab88d.gz and it looks like:
{"anonymous_id":"123","url":"-","event_type":"pageView","timestamp":"2022-07-12T03:29:47.186Z","source_ip":"69.113.177.222","user_agent":"curl/7.54.0"} {"anonymous_id":"123","url":"-","event_type":"pageView","timestamp":"2022-07-12T03:29:50.726Z","source_ip":"69.113.177.222","user_agent":"curl/7.54.0"} {"anonymous_id":"123","url":"-","event_type":"pageView","timestamp":"2022-07-12T03:29:53.628Z","source_ip":"69.113.177.222","user_agent":"curl/7.54.0"}
My question is - how come my data is automatically partitioned in Athena? When I run: select * from page_view_store_staging, it returns: my columns, plus columns for four (4) partitions 0-3 with column partition_0 having a value of 2022 etc.
I did not specify this in my config, did I?
Related
I have used the below cloudformation template code. The issue is, if I did not use 'RoleArn' under 'Targets' the script is running and there is no error but role is not attached under targets and snapshot of the ebs volume is also not created. If I include 'RoleArn' under 'Targets' Iam getting the same error as mentioned above. I have created an volume,EBSsnapshot role and Snapshot rule.
The following is the respective code:
EBSVolume:
Type: 'AWS::EC2::Volume'
Properties:
Size: 1
AvailabilityZone: ap-south-1a
EBSSnapshotRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: root
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'ec2:CreateSnapshot'
Resource: '*'
EBSSnapshotRule:
Type: 'AWS::Events::Rule'
Properties:
Description: creates a daily snapshot of EBS volume (1 a.m.)
ScheduleExpression: cron(0 1 * * ? *)
State: ENABLED
RoleArn: !GetAtt
- EBSSnapshotRole
- Arn
Name:
Ref: 'AWS::StackName'
Targets:
- Arn:
'Fn::Join':
- ''
- - 'arn:aws:automation:'
- Ref: 'AWS::Region'
- ':'
- Ref: 'AWS::AccountId'
- ':action/'
- EBSCreateSnapshot/EBSCreateSnapshot_
- Ref: 'AWS::StackName'
Input:
'Fn::Join':
- ''
- - '"'
- !Ref EBSVolume
- '"'
RoleArn: !GetAtt
- EBSSnapshotRole
- Arn
Id: EBSVolume
I have a requirement to select all the rules in AWS Config while deploying the resources in newly created account through Cloudformation. But I don't know how to select all the AWS Managed rules as in Console through Cloudformation. Any help would be very helpful.
AWSTemplateFormatVersion: 2010-09-09
Description: Enable AWS Config
Parameters:
AllSupported:
Type: String
Default: True
Description: Indicates whether to record all supported resource types.
AllowedValues:
- True
- False
IncludeGlobalResourceTypes:
Type: String
Default: True
Description: Indicates whether AWS Config records all supported global resource types.
AllowedValues:
- True
- False
ResourceTypes:
Type: List<String>
Description: A list of valid AWS resource types to include in this recording group, such as AWS::EC2::Instance or AWS::CloudTrail::Trail.
Default: <All>
DeliveryChannelName:
Type: String
Default: <Generated>
Description: The name of the delivery channel.
Frequency:
Type: String
Default: 24hours
Description: The frequency with which AWS Config delivers configuration snapshots.
AllowedValues:
- 1hour
- 3hours
- 6hours
- 12hours
- 24hours
Conditions:
IsAllSupported: !Equals
- !Ref AllSupported
- True
IsGeneratedDeliveryChannelName: !Equals
- !Ref DeliveryChannelName
- <Generated>
Mappings:
Settings:
FrequencyMap:
1hour : One_Hour
3hours : Three_Hours
6hours : Six_Hours
12hours : Twelve_Hours
24hours : TwentyFour_Hours
Resources:
ConfigBucket:
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Type: AWS::S3::Bucket
Properties:
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
ConfigBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref ConfigBucket
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: AWSConfigBucketPermissionsCheck
Effect: Allow
Principal:
Service:
- config.amazonaws.com
Action: s3:GetBucketAcl
Resource:
- !Sub "arn:${AWS::Partition}:s3:::${ConfigBucket}"
- Sid: AWSConfigBucketDelivery
Effect: Allow
Principal:
Service:
- config.amazonaws.com
Action: s3:PutObject
Resource:
- !Sub "arn:${AWS::Partition}:s3:::${ConfigBucket}/AWSLogs/${AWS::AccountId}/*"
- Sid: AWSConfigBucketSecureTransport
Action:
- s3:*
Effect: Deny
Resource:
- !Sub "arn:${AWS::Partition}:s3:::${ConfigBucket}"
- !Sub "arn:${AWS::Partition}:s3:::${ConfigBucket}/*"
Principal: "*"
Condition:
Bool:
aws:SecureTransport:
false
ConfigRecorderRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- config.amazonaws.com
Action:
- sts:AssumeRole
Path: /
ManagedPolicyArns:
- !Sub "arn:${AWS::Partition}:iam::aws:policy/service-role/AWS_ConfigRole"
ConfigRecorder:
Type: AWS::Config::ConfigurationRecorder
DependsOn:
- ConfigBucketPolicy
Properties:
RoleARN: !GetAtt ConfigRecorderRole.Arn
RecordingGroup:
AllSupported: !Ref AllSupported
IncludeGlobalResourceTypes: !Ref IncludeGlobalResourceTypes
ResourceTypes: !If
- IsAllSupported
- !Ref AWS::NoValue
- !Ref ResourceTypes
ConfigDeliveryChannel:
Type: AWS::Config::DeliveryChannel
DependsOn:
- ConfigBucketPolicy
Properties:
Name: !If
- IsGeneratedDeliveryChannelName
- !Ref AWS::NoValue
- !Ref DeliveryChannelName
ConfigSnapshotDeliveryProperties:
DeliveryFrequency: !FindInMap
- Settings
- FrequencyMap
- !Ref Frequency
S3BucketName: !Ref ConfigBucket
ConfigRuleForVolumeTags:
DependsOn: ConfigRecorder
Type: AWS::Config::ConfigRule
Properties:
InputParameters:
tag1Key: CostCenter
Scope:
ComplianceResourceTypes:
- "AWS::EC2::Volume"
Source:
Owner: AWS
SourceIdentifier: "REQUIRED_TAGS"
# Like this I need all the AWS Managed rules
You can't do this. There are no loops in cloudformation. But you could create a macro if you want such a functionality.
i am trying to deploy below stack using sam template where it supposed to deploy lambda and would add a s3 trigger, but iam getting following error
Getting ValidationError when calling the CreateChangeSet operation: Template error: instance of Fn::GetAtt references undefined resource"
i am not sure whats went wrong here to get such error
yml template
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
Environment:
Type: String
S3:
Type: String
Key:
Type: String
SecretMgr:
Type: String
Resources:
LambdaS3ToKinesis:
Type: AWS::Serverless::Function
Properties:
Handler: lambda_function.lambda_handler
Runtime: python3.7
Timeout: 60
FunctionName: !Sub "my_s3_to_kinesis"
CodeUri: ./test/src
Role: !GetAtt testKinesisRole.Arn
Description: "My lambda"
Environment:
Variables:
KINESIS_STREAM: !Sub "test_post_kinesis"
DDB_TRACKER_TABLE: my_tracker_table
ENVIRONMENT: !Sub "${Environment}"
BUCKET_NAME: !Sub "${S3}"
Events:
FileUpload:
Type: S3
Properties:
Bucket: !Sub "${S3}"
Events: s3:ObjectCreated:*
Filter:
S3Key:
Rules:
- Name: prefix
Value: "${Environment}/test1/INPUT/"
- Name: suffix
Value: ".json"
- Name: prefix
Value: "${Environment}/test2/INPUT/"
- Name: suffix
Value: ".json"
LambdaTest1KinesisToDDB:
Type: AWS::Serverless::Function
Properties:
Handler: lambda_function.lambda_handler
Runtime: python3.7
Timeout: 60
FunctionName: !Sub "${Environment}_test1_to_ddb"
CodeUri: test1_kinesis_to_ddb/src/
Role: !GetAtt testKinesisToDDBRole.Arn
Description: "test post kinesis"
Layers:
- !Ref LambdaLayertest1
Environment:
Variables:
BUCKET_NAME: !Sub "${S3}"
DDB_ACC_PLCY_TABLE:test1
DDB_TRACKER_TABLE: test_tracker
ENVIRONMENT: !Sub "${Environment}"
S3_INVALID_FOLDER_PATH: invalid_payload/
S3_RAW_FOLDER_PATH: raw_payload/
S3_UPLOAD_FLAG: false
Events:
KinesisEvent:
Type: Kinesis
Properties:
Stream: !GetAtt Kinesistest1.Arn
StartingPosition: LATEST
BatchSize: 1
Enabled: true
MaximumRetryAttempts: 0
LambdaLayerTest1KinesisToDDB:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: !Sub "${Environment}_test1_kinesis_to_ddb_layer"
ContentUri: test1_kinesis_to_ddb/dependencies/
CompatibleRuntimes:
- python3.7
Metadata:
BuildMethod: python3.7
testKinesisRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub "${Environment}_s3_to_kinesis_role"
Description: Role for first lambda
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- s3.amazonaws.com
- lambda.amazonaws.com
Action:
- "sts:AssumeRole"
Policies:
- PolicyName: !Sub "${Environment}_s3_to_kinesis_policy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:DeleteObject
Resource:
- !Sub "arn:aws:s3:::${S3}/*"
- !Sub "arn:aws:s3:::${S3}"
- Effect: Allow
Action:
- kinesis:PutRecord
Resource:
- !Sub "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:mystream1/${Environment}_test1"
- !Sub "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:mystream2/${Environment}_test2"
- Effect: Allow
Action:
- lambda:*
- cloudwatch:*
Resource: "*"
- Effect: Allow
Action:
- dynamodb:Put*
- dynamodb:Get*
- dynamodb:Update*
- dynamodb:Query
Resource:
- !GetAtt Dynamomytracker.Arn
- Effect: Allow
Action:
- kms:*
Resource:
- !Sub "${Key}"
testKinesisToDDBRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub "${Environment}_test1_to_ddb_role"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- kinesis.amazonaws.com
- lambda.amazonaws.com
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:test/service-role/AWSLambdaBasicExecutionRole"
Policies:
- PolicyName: !Sub "${Environment}_test1_kinesis_to_ddb_policy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:DeleteObject
Resource:
- !Sub "arn:aws:s3:::${S3}/*"
- !Sub "arn:aws:s3:::${S3}"
- Effect: Allow
Action:
- kinesis:Get*
- kinesis:List*
- kinesis:Describe*
Resource:
- !GetAtt KinesisTest1.Arn
- !GetAtt KinesisTest2.Arn
- Effect: Allow
Action:
- dynamodb:Put*
- dynamodb:Get*
- dynamodb:Describe*
- dynamodb:List*
- dynamodb:Update*
- dynamodb:Query
- dynamodb:DeleteItem
- dynamodb:BatchGetItem
- dynamodb:BatchWriteItem
- dynamodb:Scan
Resource:
- !Sub
- "${Table}*"
- { Table: !GetAtt "Dynamotest.Arn" }
- !Sub
- "${Table}*"
- { Table: !GetAtt "Dynamotest.Arn" }
- Effect: Allow
Action:
- kms:*
Resource:
- !Sub "${Key}"
######################################
# Update for TEst2
######################################
KinesisTest2:
Type: AWS::Kinesis::Stream
Properties:
Name: !Sub ${Environment}_test2_kinesis
StreamEncryption:
EncryptionType: KMS
KeyId: !Sub "${Key}"
RetentionPeriodHours: 24
ShardCount: 1
LambdaLayerTest2KinesisToDDB:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: !Sub "${Environment}_test2_kinesis_to_ddb_layer"
ContentUri: test2_kinesis_to_ddb/dependencies/
CompatibleRuntimes:
- python3.7
Metadata:
BuildMethod: python3.7
LambdaTest2KinesisToDDB:
Type: AWS::Serverless::Function
Properties:
Handler: lambda_function.lambda_handler
Runtime: python3.7
Timeout: 60
FunctionName: !Sub "${Environment}_Test2_kinesis_to_ddb"
CodeUri: Test2_kinesis_to_ddb/src/
Role: !GetAtt testKinesisToDDBRole.Arn
Description: "Test2"
Layers:
- !Ref LambdaLayerTest2KinesisToDDB
Environment:
Variables:
BUCKET_NAME: !Sub "${S3}"
DDB_ACC_PLCY_TABLE: my_table2
DDB_TRACKER_TABLE: my_log
ENVIRONMENT: !Sub "${Environment}"
S3_INVALID_FOLDER_PATH: invalid_payload/
S3_RAW_FOLDER_PATH: raw_payload/
S3_UPLOAD_FLAG: false
Events:
KinesisEvent:
Type: Kinesis
Properties:
Stream: !GetAtt KinesisTest2.Arn
StartingPosition: LATEST
BatchSize: 1
Enabled: true
MaximumRetryAttempts: 0
can anybody help me how can resolve this? i am not sure what exactly missed in the template and how to resolve this error
You are using AWS Serverless Application Model and your template does not conform to its format. For example, its missing required Transform statement:
Transform: AWS::Serverless-2016-10-31
There could be many other things wrong, as your template is nor CloudFormation nor Serverless at this point.
I have 2 policies each for S3 and Kinesis stream which includes DescribeStream. The S3 policy works well but I am getting this error with KinesisPolicy.
Resources:
S3
KinesisStream
Firehose
Role:
FirehoseRole
Policies:
S3 policy with the following permissions:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Kinesis Policy with the following permissions:
- 'kinesis:PutRecord'
- 'kinesis:DescribeStreamSummary'
- 'kinesis:PutRecords'
- 'kinesis:GetShardIterator'
- 'kinesis:GetRecords'
- 'kinesis:DescribeStream'
Error:
The role (firehoseRole) is not authorized to perform DescribeStream on MyKinesisStream.
Cloud formation template
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
VersioningConfiguration:
Status: Enabled
firehoseRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: 'sts:AssumeRole'
Condition:
StringEquals:
'sts:ExternalId': !Ref 'AWS::AccountId'
DeliveryPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: firehose_delivery_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Resource:
- !Sub 'arn:aws:s3:::${S3Bucket}'
- !Sub 'arn:aws:s3:::${S3Bucket}*'
Roles:
- !Ref firehoseRole
KinesisPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: kinesis_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'kinesis:PutRecord'
- 'kinesis:DescribeStreamSummary'
- 'kinesis:PutRecords'
- 'kinesis:GetShardIterator'
- 'kinesis:GetRecords'
- 'kinesis:DescribeStream'
Resource:
- !GetAtt MyKinesisStream.Arn
Roles:
- !Ref firehoseRole
MyKinesisStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 1
DeliveryStream:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
DeliveryStreamType: KinesisStreamAsSource
KinesisStreamSourceConfiguration:
KinesisStreamARN: !GetAtt MyKinesisStream.Arn
RoleARN: !GetAtt firehoseRole.Arn
S3DestinationConfiguration:
BucketARN: !GetAtt S3Bucket.Arn
BufferingHints:
IntervalInSeconds: 60
SizeInMBs: 50
CompressionFormat: UNCOMPRESSED
Prefix: firehose/
RoleARN: !GetAtt firehoseRole.Arn
I was able to resolve the error. I had to add DependsOn To DeliveryStream and include both the policies.
I've made a parent (nested) stack template that references 4 child templates. When I launch the stack through aws cloudformation create-stack, I get the following error for the parent stack:
Embedded stack AlignmentLambdaFunction was not successfully created: The following resource(s) failed to create: [CloudspanLambdaFunction, HaploLambdaExecutionRole, AlignmentLambdaExecutionRole].
And I get this error within one of the nested stacks that was getting created from the parent: Policy contains a statement with one or more invalid principals (for MasterGCPStorageKey (which is a resource in the Lambda child above)
I don't understand the source of the error. I thought maybe it was because of needing a DependsOn for the ExecutionRoles, but that didn't resolve the error.
Parent Stack:
AWSTemplateFormatVersion: "2010-09-09"
Description: "Master template for wgs-pipeline. Calls to other stack templates."
Parameters:
CloudspanLambdaFuncS3BucketName:
Type: String
CloudspanLambdaFuncS3KeyName:
Default: 'sfn.deployable.zip'
Type: String
CloudspanLambdaFuncModuleName:
Default: 'cloudspan'
Type: String
AlignmentLambdaFuncS3BucketName:
Type: String
AlignmentLambdaFuncS3KeyName:
Type: String
AlignmentLambdaFuncModuleName:
Type: String
HaploLambdaFuncS3BucketName:
Type: String
HaploLambdaFuncS3KeyName:
Type: String
HaploLambdaFuncModuleName:
Type: String
KMSAdminUserARN:
Type: String
KMSEndUserARN:
Type: String
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
InternetGateway:
Type: AWS::EC2::InternetGateway
RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId:
Ref: 'VPC'
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId:
Ref: 'VPC'
InternetGatewayId:
Ref: 'InternetGateway'
SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: EC2 Security Group for instances launched in the VPC by Batch
VpcId:
Ref: 'VPC'
StepFunctionsActivitiesInstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allow http to client host
VpcId:
Ref: VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp: 128.218.0.0/16
Subnet:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.0.0/24
VpcId:
Ref: 'VPC'
AvailabilityZone:
Ref: GPCESubnetAZ1
MapPublicIpOnLaunch: 'True'
DependsOn: VPC
Route:
Type: AWS::EC2::Route
Properties:
RouteTableId:
Ref: 'RouteTable'
DestinationCidrBlock: 0.0.0.0/0
GatewayId:
Ref: 'InternetGateway'
DependsOn:
- RouteTable
- InternetGateway
SubnetRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId:
Ref: 'RouteTable'
SubnetId:
Ref: 'Subnet'
DependsOn:
- RouteTable
- Subnet
# Beginning of reference to child stacks
ClouspanLambdaFunction:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
CloudspanLambdaFuncS3BucketName:
Ref: CloudspanLambdaFuncS3BucketName
CloudspanLambdaFuncS3KeyName:
Ref: CloudspanLambdaFuncS3KeyName
CloudspanLambdaFuncModuleName:
Ref: CloudspanLambdaFuncModuleName
KMSAdminUserARN:
Ref: KMSAdminUserARN
KMSEndUserARN:
Ref: KMSEndUserARN
TemplateURL: https://s3.amazonaws.com/CFNTemplate/lambda_resources.stack.yaml
TimeoutInMinutes: 1
AlignmentLambdaFunction:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
AlignmentLambdaFuncS3BucketName:
Ref: AlignmentLambdaFuncS3BucketName
AlignmentLambdaFuncS3KeyName:
Ref: AlignmentLambdaFuncS3KeyName
AlignmentLambdaFuncModuleName:
Ref: AlignmentLambdaFuncModuleName
KMSAdminUserARN:
Ref: KMSAdminUserARN
KMSEndUserARN:
Ref: KMSEndUserARN
TemplateURL: https://s3.amazonaws.com/CFNTemplate/lambda_resources.stack.yaml
TimeoutInMinutes: 1
HaploLambdaFunction:
Type: "AWS::CloudFormation::Stack"
Properties:
Parameters:
HaploLambdaFuncS3BucketName:
Ref: HaploLambdaFuncS3BucketName
HaploLambdaFuncS3KeyName:
Ref: HaploLambdaFuncS3KeyName
HaploLambdaFuncModuleName:
Ref: HaploLambdaFuncModuleName
KMSAdminUserARN:
Ref: KMSAdminUserARN
KMSEndUserARN:
Ref: KMSEndUserARN
TemplateURL: https://s3.amazonaws.com/CFNTemplate/lambda_resources.stack.yaml
TimeoutInMinutes: 1
Lambda Child Stack (relevant for error):
AWSTemplateFormatVersion: '2010-09-09'
Description: lambda function and execution role stack.
Parameters:
CloudspanLambdaFuncS3BucketName:
Type: String
Default: 'claudia-test-transfer'
CloudspanLambdaFuncS3KeyName:
Default: 'sfn.deployable.zip'
Type: String
CloudspanLambdaFuncModuleName:
Default: 'cloudspan'
Type: String
AlignmentLambdaFuncS3BucketName:
Type: String
Default: 'claudia-test-transfer'
AlignmentLambdaFuncS3KeyName:
Type: String
Default: 'alignment_processing.deployable.zip'
AlignmentLambdaFuncModuleName:
Type: String
Default: 'alignment_processing'
HaploLambdaFuncS3BucketName:
Type: String
Default: 'claudia-test-transfer'
HaploLambdaFuncS3KeyName:
Type: String
Default: 'sentieon_haplotyper.deployable.zip'
HaploLambdaFuncModuleName:
Type: String
Default: 'sentieon_haplotyper'
KMSAdminUserARN:
Type: String
KMSEndUserARN:
Type: String
Resources:
CloudspanLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Handler:
Fn::Join: [ ".", [ Ref: CloudspanLambdaFuncModuleName, "handler"] ]
Role:
Fn::GetAtt: [ CloudspanLambdaExecutionRole, Arn ]
Code:
S3Bucket:
Ref: CloudspanLambdaFuncS3BucketName
S3Key:
Ref: CloudspanLambdaFuncS3KeyName
Runtime: "python3.6"
Timeout: "60"
DependsOn: CloudspanLambdaExecutionRole
AlignmentLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Handler:
Fn::Join: [ ".", [ Ref: AlignmentLambdaFuncModuleName, "handler"] ]
Role:
Fn::GetAtt: [ AlignmentLambdaExecutionRole, Arn ]
Code:
S3Bucket:
Ref: AlignmentLambdaFuncS3BucketName
S3Key:
Ref: AlignmentLambdaFuncS3KeyName
Runtime: "python3.6"
Timeout: "60"
DependsOn: AlignmentLambdaExecutionRole
HaploLambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Handler:
Fn::Join: [ ".", [ Ref: HaploLambdaFuncModuleName, "handler"] ]
Role:
Fn::GetAtt: [ HaploLambdaExecutionRole, Arn ]
Code:
S3Bucket:
Ref: HaploLambdaFuncS3BucketName
S3Key:
Ref: HaploLambdaFuncS3KeyName
Runtime: "python3.6"
Timeout: "60"
DependsOn: HaploLambdaExecutionRole
CloudspanLambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: "sts:AssumeRole"
Policies:
- PolicyName: CanListBuckets
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "s3:GetBucketLocation"
- "s3:ListAllMyBuckets"
Resource: "arn:aws:s3:::*"
- PolicyName: CanLog
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
AlignmentLambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: "sts:AssumeRole"
Policies:
- PolicyName: CanListBuckets
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "s3:GetBucketLocation"
- "s3:ListAllMyBuckets"
Resource: "arn:aws:s3:::*"
- PolicyName: CanCallBatch
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "batch:*"
Resource: "*"
- PolicyName: CanLog
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
HaploLambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: "sts:AssumeRole"
Policies:
- PolicyName: CanListBuckets
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "s3:GetBucketLocation"
- "s3:ListAllMyBuckets"
Resource: "arn:aws:s3:::*"
- PolicyName: CanCallBatch
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "batch:*"
Resource: "*"
- PolicyName: CanLog
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
MasterGCPStorageKey:
Type: "AWS::KMS::Key"
Properties:
Description: Symmetric Master Key for GCP Storage Credentials off-line encryption/on-line decryption protocol
Enabled: True
EnableKeyRotation: True
KeyPolicy:
Version: "2012-10-17"
Statement:
- Sid: "Allow Lambda Excution Role access to GCP Storage decryption key"
Effect: "Allow"
Principal:
# ARN of CloudspanLambdaExecutionRole
AWS:
Fn::GetAtt: [ CloudspanLambdaExecutionRole, Arn ]
Action:
- kms:Decrypt
- kms:DescribeKey
# in this context "*" means "this" CMK
Resource: "*"
- Sid: "Allow Administrator to admin the GCP Storage decryption key"
Effect: "Allow"
Principal:
# ARN of the KMS admin IAM user
AWS:
Ref: KMSAdminUserARN
Action:
- "kms:Create*"
- "kms:Describe*"
- "kms:Enable*"
- "kms:List*"
- "kms:Put*"
- "kms:Update*"
- "kms:Revoke*"
- "kms:Disable*"
- "kms:Get*"
- "kms:Delete*"
- "kms:TagResource"
- "kms:UntagResource"
- "kms:ScheduleKeyDeletion"
- "kms:CancelKeyDeletion"
- "kms:Encrypt"
- "kms:Decrypt"
- "kms:ReEncrypt"
- "kms:GenerateDataKey*"
- "kms:DescribeKey"
# in this context "*" means "this" CMK
Resource: "*"
- Sid: "Allow End User to encrypt the GCP Storage creds"
Effect: "Allow"
Principal:
# ARN of the KMS IAM end user
AWS:
Ref: KMSEndUserARN
Action:
- "kms:Encrypt"
- "kms:ReEncrypt"
- "kms:DescribeKey"
# in this context "*" means "this" CMK
Resource: "*"
DependsOn: CloudspanLambdaExecutionRole
I also was getting the following error after re-deploying a CloudFormation stack I had removed (via Serverless):
We encountered the following errors while processing your request:
Policy contains a statement with one or more invalid principals.
In my case, the original role which was assigned to my KMS encryption key was removed. KMS still keeps a reference to the removed role, and apparently adding a newly created role of the same type creates this error.
I solved this by simply removing the old reference to the removed role, under IAM > Encryption Keys > YOUR_KEY_NAME > Key Policy > Key Users