I am trying to access RDS mysql database via lambda function. I am deploying as SAM template. I have a lambda function attached to an execution role as the following:
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
Path: "/"
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- rds:*
Resource: "*"
CreateTaskFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./components/lambdaFunctions/createTask
Handler: createTask.handler
Runtime: nodejs12.x
Role: !GetAtt LambdaExecutionRole.Arn
Timeout: 500
Events:
ProxyApiRoot:
Type: Api
Properties:
RestApiId: !Ref ApiGatewayApi
Path: /
Method: ANY
ProxyApiGreedy:
Type: Api
Properties:
RestApiId: !Ref ApiGatewayApi
Path: /{proxy+}
Method: ANY
Layers:
- !Ref NodeModulesLayer
After deploying the stack the lambda can't connect to RDS, and I found only the cloudwatch logs roles in the permission section of lambda:
As you see the RDS permission is not listed. Any suggestions?
Oh my bad. I figured it out, it was a VPC issue. Lambda has to be attached to a VPC, and a security group that is allowed by the security group of the database.
Related
I couldn't get a valid IAM policy to work for a Lambda function to OpenSearch.
Replicate:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ${AWS::StackName}-Replicate
Description: !Sub
- Stack ${StackTagName} Environment ${EnvironmentTagName} Function ${ResourceName}
- ResourceName: DBReplicate
CodeUri: ../src/Replicate
Handler: index.handler
Runtime: nodejs16.x
MemorySize: 3008
Timeout: 30
Tracing: Active
Policies:
- PolicyName: Access
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- es:*
Resource:
- arn:aws:es:eu-west-1:22222222222:domain/mynewdomain
- DomainName: mynewdomain
Events:
MyDynamoDBtable:
Type: DynamoDB
Properties:
Stream: !Ref TableStreamArn
StartingPosition: TRIM_HORIZON
BatchSize: 1
Running sam validate, I'm getting:
Policy at index 0 in the 'Policies' property is not valid
Got the answer from another post, the structure was wrong, the correct way is:
Add inline policy to aws SAM template
QueryFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: query/
Handler: app.lambda_handler
Policies:
- AmazonDynamoDBFullAccess
- AWSLambdaVPCAccessExecutionRole
- Version: '2012-10-17' # Policy Document
Statement:
- Effect: Allow
Action:
- dynamodb:*
Resource: 'arn:aws:dynamodb:*:*:table/dynamo_db_table_endpoint'
I'm trying to setup an S3 bucket which notifies a Lambda function when a new object is created.
The stack below works fine but I want to added the SourceArn to the Lambda permission, following best practices.
There's some literature on this which suggests the way to do it is via a string rather than Fn::GetAtt/Arn -
https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-circular-dependency-cloudformation/
But if I uncomment the relevant SourceArn line and redeploy, I get -
Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: TBZDEVZ1HVD9EN0Q; S3 Extended Request ID: gL3CRz6UayvHup5i5oC4+/RMm0p1oRaRrPVtfZrykeAaJ1BVuhNSKkqxQ8TL5sy749d9PtbMOEQ=; Proxy: null)
Ho hum. Looking at the article again, I see that MyBucket needs to depend on MyBucketFunctionPermission - but if I uncomment that and redeploy I now get -
An error occurred (ValidationError) when calling the CreateChangeSet operation: Circular dependency between resources: [MyBucketFunctionPermission, MyBucket]
This is some fresh circle of hell. Am I missing something from the article or is there some other combination of SourceArn format + DependsOn that would get this to work ?
TIA.
AWSTemplateFormatVersion: '2010-09-09'
Outputs: {}
Parameters: {}
Resources:
MyBucket:
# DependsOn:
# - MyBucketFunctionPermission
Properties:
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function:
Fn::GetAtt:
- MyBucketFunction
- Arn
Type: AWS::S3::Bucket
MyBucketFunction:
Properties:
Code:
ZipFile: "def handler(event, context):\n print (event)"
Handler: index.handler
Role:
Fn::GetAtt:
- MyBucketRole
- Arn
Runtime: "python3.8"
Type: AWS::Lambda::Function
MyBucketFunctionPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: MyBucketFunction
Principal: s3.amazonaws.com
# SourceArn:
# Fn::Sub: arn:aws:s3:::${MyBucket}
Type: AWS::Lambda::Permission
MyBucketRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: logs:*
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: my-bucket-role-policy-1234567890
Type: AWS::IAM::Role
There is nothing seemingly wrong with your template. It works fine, at least for me. But a race condition is possible between MyBucket and MyBucketFunctionPermission. Thus, protect against this, you have to use DependsOn. But for that to work you have to explicitly set your bucket name. For example:
AWSTemplateFormatVersion: '2010-09-09'
Outputs: {}
Parameters: {}
Resources:
MyBucket:
DependsOn:
- MyBucketFunctionPermission
Properties:
BucketName: !Sub "my-bucket-323323-${AWS::StackName}-${AWS::Region}"
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function:
Fn::GetAtt:
- MyBucketFunction
- Arn
Type: AWS::S3::Bucket
MyBucketFunction:
Properties:
Code:
ZipFile: "def handler(event, context):\n print (event)"
Handler: index.handler
Role:
Fn::GetAtt:
- MyBucketRole
- Arn
Runtime: "python3.8"
Type: AWS::Lambda::Function
MyBucketFunctionPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: MyBucketFunction
Principal: s3.amazonaws.com
SourceArn: !Sub "arn:aws:s3:::my-bucket-323323-${AWS::StackName}-${AWS::Region}"
Type: AWS::Lambda::Permission
MyBucketRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: logs:*
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: my-bucket-role-policy-1234567890
Type: AWS::IAM::Role
I want to create a lambda function through cloudformation template, I have ConfigurationLambdaRole, ConfigurationLambdaFunction and ConfigurationLambdaInvokePermission, in ConfigurationLambdaInvokePermission section, what should be the SourceArn? And is there any incorrect thing with my template?
Resources:
ConfigurationLambdaRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: 'configuration-lambda'
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
- s3.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSQSFullAccess
- arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
ConfigurationLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Description: 'configuration service with lambda'
FunctionName: 'configuration-lambda1'
Handler: lambda.handler.EventHandler::handleRequest
Runtime: Java 11
MemorySize: 128
Timeout: 120
Code:
S3Bucket: configurationlambda
S3Key: lambda-service-1.0.0-SNAPSHOT.jar
Role: !GetAtt ConfiguratioLambdaRole.Arn
ConfigurationLambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Fn::GetAtt:
- ConfigurationLambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: "s3.amazonaws.com"
SourceArn: 'arn of jar file in s3(configurationlambda)'
SourceArn is an arn of a resource which is going to invoke your function. For example, if your lambda would be invoked through S3 Events Notifications, the SourceArn would be the ARN of your bucket.
In your case I don't see why would you need AWS::Lambda::Permission. So I would just remove the entire resource.
I have to add retention policy to API Gateway Cloudwatch logs, hence I cannot use the aws provided policy to do so i.e. arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
So instead I created my own role with custom policy :
ApiGatewayCloudWatchLogsRole:
Type: 'AWS::IAM::Role'
DependsOn: APIGFunctionLogGroup
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- apigateway.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: APIGatewayPushLogsPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
Effect: Allow
Action:
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 'logs:DescribeLogGroups'
- 'logs:DescribeLogStreams'
- 'logs:GetLogEvents'
- 'logs:FilterLogEvents'
Resource: '*'
And then created LogGroup with retention as :
APIGFunctionLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
RetentionInDays: 30
LogGroupName: !Join
- ''
- - API-Gateway-Execution-Logs_
- !Ref MyRestApi
And passed the above created role to AWS::ApiGateway::Account
ApiGatewayAccount:
Type: 'AWS::ApiGateway::Account'
DependsOn: APIGFunctionLogGroup
Properties:
CloudWatchRoleArn: !GetAtt
- ApiGatewayCloudWatchLogsRole
- Arn
But while deploying my API Gateway I am getting error as :
I have the trust policy as well but API Gateway Account is not getting created.
If you create the log group yourself, before APIgateway does you should be able to use the existing policy/service role.
I am looking for a Cloudformation template to push cloudwatch logs to elasticsearch in another account. Even to the same account would be ok and I can update that.
Seems like a standard problem but haven't seen any template which automates the steps described in https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html.
I should have updated the answer. This is what I wrote which is the minimalistic version of streaming logs from CW to Elastic Search.
Resources:
LambdaElasticSearchExecutionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action: ['es:ESHttpPost']
Effect: Allow
Resource: "*"
- Action: ['logs:CreateLogGroup', 'logs:CreateLogStream', 'logs:PutLogEvents']
Effect: Allow
Resource: "arn:aws:logs:*:*:*"
- Action: ['lambda:InvokeFunction']
Effect: Allow
Resource: "arn:aws:logs:*:*:*"
Version: '2012-10-17'
PolicyName: lambdaRoleElasticSearchStreaming
Type: AWS::IAM::Role
ESStreamingLambda:
Type: AWS::Lambda::Function
DependsOn: LambdaElasticSearchExecutionRole
Properties:
Handler: index.handler
Role:
Fn::GetAtt: [LambdaElasticSearchExecutionRole, Arn]
Code:
S3Bucket: {'Fn::Sub': 'do-not-delete-cw-es-log-streaming-lamda-${AWS::Region}'}
S3Key: LogsToElasticsearch.zip
Runtime: nodejs4.3
LambdaPermissionForCWInvokation:
DependsOn: ESStreamingLambda
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName:
Fn::GetAtt: [ESStreamingLambda, Arn]
Principal: {'Fn::Sub': 'logs.${AWS::Region}.amazonaws.com'}
APILogsToElasticSearchSubscriptionFilter:
Type: AWS::Logs::SubscriptionFilter
DependsOn: [ESStreamingLambda, LambdaPermissionForCWInvokation]
Properties:
DestinationArn:
Fn::GetAtt: [ESStreamingLambda, Arn]
FilterPattern: ''
LogGroupName: {Ref: LambdaLogGroup}
```java
I am currently trying to do the same thing. I have found these examples for now:
https://docs.aws.amazon.com/solutions/latest/centralized-logging/templates.html (seems to be the best bet)
https://github.com/amazon-archives/cloudwatch-logs-subscription-consumer (seems to me this is an old approach)
https://github.com/awslabs/hids-cloudwatchlogs-elasticsearch-template (just an example)