Im new to aws and was trying to use Event brigde and api destination for pinging and endpoint. I am successful in creating api destination and api connection
using cloud formation template and also successfully created IAM role and rule for the same, but i want to have the logs in cloudwatch triggering that endpoint after a scheduled expression. (There is no event triggering that endpoint, i want to trigger it after every 5 min)
Below is my cloud formation template
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
Action:
- sts:AssumeRole
Path: '/jobs-role/'
Policies:
- PolicyName: eventbridge-api-destinations
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- events:InvokeApiDestination
Resource: !GetAtt Destination.Arn
ApiConnection:
Type: AWS::Events::Connection
Properties:
AuthorizationType: API_KEY
Description: Connection to API
AuthParameters:
ApiKeyAuthParameters:
ApiKeyName: authorization
ApiKeyValue: MyAPIkey
Destination:
Type: AWS::Events::ApiDestination
Properties:
ConnectionArn: !GetAtt ApiConnection.Arn
Description: API Destination to send events
HttpMethod: POST
InvocationEndpoint: !Ref ApiDestinationInvocationEndpoint
InvocationRateLimitPerSecond: 10
RecordsRule:
Type: AWS::Events::Rule
Properties:
Description: !Sub 'Trigger ${RecordsRole} according to the specified schedule'
State: ENABLED
ScheduleExpression: "rate(5 minutes)"
Targets:
- Id: !Sub '${RecordsRole}'
Arn: !GetAtt Destination.Arn
RoleArn: !GetAtt RecordsRole.Arn
HttpParameters:
HeaderParameters:
Content-type: application/json;charset=utf-8
Can anyone tell me what i am lacking?
Below is link to my screenshot of invocations graph in metrics
[1]: https://i.stack.imgur.com/Zms44.png
Related
I'm following this tutorial on enabling aws security hub with aws chatbot. I seem to be having an issue with deploying the custom resource named AddCustomActionSHResource1 in the following code, which is linked from the tutorial to here.
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys CustomAction in SecurityHub to enable sending findings to Slack via AWS Chatbot
#==================================================
# Parameters
#==================================================
Parameters:
SlackWorkSpaceID:
Description: Slack workspace ID (Copy and Paste from AWS Chatbot Configured Clients Interface)
Type: String
MinLength: 9
MaxLength: 15
AllowedPattern: ^[a-zA-Z0-9_]*$
ConstraintDescription: |
Malformed Input Parameter: Environment must contain only upper and numbers. Length should be minimum of 9 characters and a maximum of 15 characters.
SlackChannelID:
Description: Slack Channel ID
Type: String
MinLength: 9
MaxLength: 15
AllowedPattern: ^[a-zA-Z0-9_]*$
ConstraintDescription: |
Malformed Input Parameter: Environment must contain only upper and numbers. Length should be a minimum of 9 characters and a maximum of 15 characters.
# CustomActionName:
# Description: Name of the Custom Action in SecurityHub
# Type: String
# AllowedPattern: ^[a-zA-Z0-9_]*$
# Default: Send_To_Slack
# #Default: 'Send To !Sub "${ChatApplication}"'
#==================================================
# Resources
#==================================================
Resources:
#======================================================
# Lambda Role to create Custom Action
#======================================================
LambdaIAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSLambdaExecute
Path: /
#======================================================
# Lambda Policy to create Custom Action
#======================================================
LambdaIAMPolicy:
Type: AWS::IAM::Policy
DependsOn: LambdaIAMRole
Properties:
PolicyName: LambdaCreateCustomActionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'securityhub:CreateActionTarget'
Resource: '*'
Roles:
- !Ref LambdaIAMRole
#======================================================
# Lambda Function to create Custom Action
#======================================================
LambdaCreateCustomAction:
Type: AWS::Lambda::Function
DependsOn: LambdaIAMPolicy
Properties:
FunctionName: addcustomactionsecurityhub
Description: CreateCustom Action in SecurityHub
Runtime: python3.7
Handler: index.lambda_handler
Code:
ZipFile: |
import boto3
import cfnresponse
def lambda_handler(event, context):
securityhub = boto3.client('securityhub')
response = securityhub.create_action_target(Name="Send_To_Slack",Description='Send Messages to ChatApplication via AWS ChatBot',Id='SendToSlack')
responseData = {}
responseData['Data'] = response
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")
MemorySize: 128
Timeout: 10
Role: !GetAtt LambdaIAMRole.Arn
#======================================================
# Custom Resource to Invoke the Lambda Function
#======================================================
AddCustomActionSHResource1:
Type: Custom::AddCustomActionSH
DependsOn: LambdaCreateCustomAction
Properties:
ServiceToken: !GetAtt LambdaCreateCustomAction.Arn
#======================================================
# SNS Topic
#======================================================
SNSTopicAWSChatBot:
Type: AWS::SNS::Topic
Properties:
DisplayName: AWS Chatbot SNS Topic
EventTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: topicPolicyCustomaction
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sns:Publish'
Resource: '*'
Topics:
- !Ref SNSTopicAWSChatBot
#======================================================
# CloudWatch Event Rule
#======================================================
EventRuleCustomAction:
Type: AWS::Events::Rule
Properties:
Description: "SecurityHub Chatbot CustomAction"
EventPattern:
source:
- "aws.securityhub"
detail-type:
- "Security Hub Findings - Custom Action"
resources:
- !Sub 'arn:aws:securityhub:${AWS::Region}:${AWS::AccountId}:action/custom/SendToSlack'
State: "ENABLED"
Targets:
-
Arn:
Ref: "SNSTopicAWSChatBot"
Id: "OpsTopic"
ChatBotManageIAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "sns.amazonaws.com"
Action:
- "sts:AssumeRole"
SlackChannelConfig:
Type: AWS::Chatbot::SlackChannelConfiguration
Properties:
ConfigurationName: securityhubnotification
IamRoleArn: !GetAtt ChatBotManageIAMRole.Arn
LoggingLevel: NONE
SlackChannelId: !Ref SlackChannelID
SlackWorkspaceId: !Ref SlackWorkSpaceID
SnsTopicArns:
- !Ref SNSTopicAWSChatBot
When deploying this cloudformation file to cloudformation in AWS, everything deploys successfully except for AddCustomActionSHResource1. The error message is:
CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [18sa90d1-49s1-4as7-9fsc-b79ssd6csd9]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
I found some information on updating the lambda function by just adding comments to it, but I don't believe that's the issue. Most of the useful information I found was from this aws link.
I am trying to access RDS mysql database via lambda function. I am deploying as SAM template. I have a lambda function attached to an execution role as the following:
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
Path: "/"
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- rds:*
Resource: "*"
CreateTaskFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: ./components/lambdaFunctions/createTask
Handler: createTask.handler
Runtime: nodejs12.x
Role: !GetAtt LambdaExecutionRole.Arn
Timeout: 500
Events:
ProxyApiRoot:
Type: Api
Properties:
RestApiId: !Ref ApiGatewayApi
Path: /
Method: ANY
ProxyApiGreedy:
Type: Api
Properties:
RestApiId: !Ref ApiGatewayApi
Path: /{proxy+}
Method: ANY
Layers:
- !Ref NodeModulesLayer
After deploying the stack the lambda can't connect to RDS, and I found only the cloudwatch logs roles in the permission section of lambda:
As you see the RDS permission is not listed. Any suggestions?
Oh my bad. I figured it out, it was a VPC issue. Lambda has to be attached to a VPC, and a security group that is allowed by the security group of the database.
Cloudtrail default logs can be streamed to elasticsearch domain as shown in this image. How do I achieve this using cloudformation template?
Update:
If you are using aws-cli, take a look at my answer here.
Well, after a few hours of exploring and reading a lot of documentation I finally succeeded to create this template.
Designer Overview :
In order to enable the stream logs to elasticsearch we need to create the following resources:
The lambda function will forward the logs from cloudwatch log group to Elasticsearch.
Relevant IAM Role to get logs from cloudwatch and insert to Elasticsearch.
Lambda permission - The AWS::Lambda::Permission resource grants an AWS service or another account permission to use a function to allow the cloudwatch log group to trigger the lambda.
Subscription Filter - The AWS::Logs::SubscriptionFilter resource specifies a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events and have them delivered to a specific destination.
Template usage:
Download LogsToElasticsearch.zip from my Github page.
Update var endpoint = '${Elasticsearch_Endpoint}'; in index.js with your Elasticseatch url e.g - 'search-xxx-yyyy.eu-west-1.es.amazonaws.com';.
Copy the zip file to s3 bucket which will be used in the template (LambdaArtifactBucketName).
Fill relevant Parameters - you can find descriptions to each resource.
Template YAML:
AWSTemplateFormatVersion: 2010-09-09
Description: Enable logs to elasticsearch
Parameters:
ElasticsearchDomainName:
Description: Name of the Elasticsearch domain that you want to insert logs to
Type: String
Default: amitb-elastic-domain
CloudwatchLogGroup:
Description: Name of the log group you want to subscribe
Type: String
Default: /aws/eks/amitb-project/cluster
LambdaName:
Description: Name of the lambda function
Type: String
Default: amitb-cloudwatch-logs
LambdaRole:
Description: Name of the role used by the lambda function
Type: String
Default: amit-cloudwatch-logs-role
LambdaArtifactBucketName:
Description: The bucket where the lambda function located
Type: String
Default: amit-bucket
LambdaArtifactName:
Description: The name of the lambda zipped file
Type: String
Default: LogsToElasticsearch.zip
VPC:
Description: Choose which VPC the Lambda-functions should be deployed to
Type: 'AWS::EC2::VPC::Id'
Default: vpc-1111111
Subnets:
Description: Choose which subnets the Lambda-functions should be deployed to
Type: 'List<AWS::EC2::Subnet::Id>'
Default: 'subnet-123456789,subnet-123456456,subnet-123456741'
SecurityGroup:
Description: Select the Security Group to use for the Lambda-functions
Type: 'List<AWS::EC2::SecurityGroup::Id>'
Default: 'sg-2222222,sg-12345678'
Resources:
ExampleInvokePermission:
Type: 'AWS::Lambda::Permission'
DependsOn: ExampleLambdaFunction
Properties:
FunctionName:
'Fn::GetAtt':
- ExampleLambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: !Sub 'logs.${AWS::Region}.amazonaws.com'
SourceArn: !Sub >-
arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CloudwatchLogGroup}:*
SourceAccount: !Ref 'AWS::AccountId'
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Ref LambdaRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: lambda-to-es-via-vpc-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'es:*'
Resource:
- !Sub >-
arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ElasticsearchDomainName}
- PolicyName: logs-and-ec2-permissions
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'ec2:CreateNetworkInterface'
- 'ec2:DescribeNetworkInterfaces'
- 'ec2:DeleteNetworkInterface'
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
ExampleLambdaFunction:
Type: 'AWS::Lambda::Function'
DependsOn: LambdaExecutionRole
Properties:
Code:
S3Bucket: !Ref LambdaArtifactBucketName
S3Key: !Ref LambdaArtifactName
FunctionName: !Ref LambdaName
Handler: !Sub '${LambdaName}.handler'
Role:
'Fn::GetAtt':
- LambdaExecutionRole
- Arn
Runtime: nodejs8.10
Timeout: '300'
VpcConfig:
SecurityGroupIds: !Ref SecurityGroup
SubnetIds: !Ref Subnets
MemorySize: 512
SubscriptionFilter:
Type: 'AWS::Logs::SubscriptionFilter'
DependsOn: ExampleInvokePermission
Properties:
LogGroupName: !Ref CloudwatchLogGroup
FilterPattern: '[host, ident, authuser, date, request, status, bytes]'
DestinationArn:
'Fn::GetAtt':
- ExampleLambdaFunction
- Arn
Results:
Cloudwatch log:
Hope you find it helpfull.
Update 02/09/2020:
node.js 8.10 is now deprecated, you should use node.js 10 or 12.
https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html
I have to add retention policy to API Gateway Cloudwatch logs, hence I cannot use the aws provided policy to do so i.e. arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
So instead I created my own role with custom policy :
ApiGatewayCloudWatchLogsRole:
Type: 'AWS::IAM::Role'
DependsOn: APIGFunctionLogGroup
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- apigateway.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: APIGatewayPushLogsPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
Effect: Allow
Action:
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 'logs:DescribeLogGroups'
- 'logs:DescribeLogStreams'
- 'logs:GetLogEvents'
- 'logs:FilterLogEvents'
Resource: '*'
And then created LogGroup with retention as :
APIGFunctionLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
RetentionInDays: 30
LogGroupName: !Join
- ''
- - API-Gateway-Execution-Logs_
- !Ref MyRestApi
And passed the above created role to AWS::ApiGateway::Account
ApiGatewayAccount:
Type: 'AWS::ApiGateway::Account'
DependsOn: APIGFunctionLogGroup
Properties:
CloudWatchRoleArn: !GetAtt
- ApiGatewayCloudWatchLogsRole
- Arn
But while deploying my API Gateway I am getting error as :
I have the trust policy as well but API Gateway Account is not getting created.
If you create the log group yourself, before APIgateway does you should be able to use the existing policy/service role.
I have a lambda function that will handle PUT and GET requests using Amazon API Gateway {proxy+}.
It is working correctly when all the settings are set manually by the Amazon Console. but I want to automate it using AWS Cloudformation.
To inform you, I will write steps to set {proxy+}:
1) create a simple Lambda function and paste this lines of code inside it:
import boto3
def lambda_handler(event, context):
return {
"statusCode": 200,
"headers": {
"Content-Type": 'text/html',
"Access-Control-Allow-Origin": "*"
},
"body": "Hello Reza Amya, Your Lambda is working..!"
}
2) goto Amazon API Gateway and click on Create API.
3) choose New API, fill API name, select Edge optimized from the list for Endpoint Type then click on Create API
4) then your API is created and you should be on it's Resources page, if you are not, go to the Resources page for the created API.
5) from Actions select Create Resource
6) Select Configure as proxy resource (then it should change other fields automatically, if it doesn't, type proxy for Resource Name and {proxy+} for Resource Path) then click on Create Resource
7) Select Lambda Function Proxy for Integration type and select your lambda function from Lambda Function and click on Save
8) on the Add Permission to Lambda Function popup, click on Ok
9) from Actions click on Deploy API
10) Select New Stage from the list for Deployment stage then type a name for Stage name (for me, I have typed 'api') and click on Deploy
11) on the stage on the root page for your deployed API, you can see Invoke URL. click on it, and it will open new tab linked to somewhere like this: https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com/api/
12) add a simple segment to end of your URL like this:
https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com/api/test
now you should see bellow message in your browser page:
Hello Reza Amya, Your Lambda is working..!
Now the problem is I have written all these steps inside a Yaml file:
AWSTemplateFormatVersion: 2010-09-09
Description: My Lambda Function
Parameters:
S3Bucket:
Description: S3 Bucket where the Lambda code is
Type: String
S3Key:
Description: S3 Key where the Lambda code is
Type: String
S3ObjectVersion:
Description: Version of the S3 Key to use
Type: String
Resources:
apiGateway:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "my-api"
Description: "My API"
EndpointConfiguration:
Types:
- EDGE
Resource:
Type: AWS::ApiGateway::Resource
Properties:
RestApiId:
Ref: "apiGateway"
ParentId:
Fn::GetAtt:
- "apiGateway"
- "RootResourceId"
PathPart: "{proxy+}"
ProxyMethod:
Type: 'AWS::ApiGateway::Method'
Properties:
HttpMethod: ANY
ResourceId: !Ref Resource
RestApiId: !Ref apiGateway
AuthorizationType: NONE
RequestParameters:
method.request.path.proxy: true
Integration:
CacheKeyParameters:
- 'method.request.path.proxy'
RequestParameters:
integration.request.path.proxy: 'method.request.path.proxy'
Type: AWS_PROXY
IntegrationHttpMethod: ANY
Uri: !Sub
- arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Arn}/invocations
- Arn:
Fn::GetAtt:
- LambdaFunction
- Arn
PassthroughBehavior: WHEN_NO_MATCH
IntegrationResponses:
- StatusCode: 200
apiGatewayDeployment:
Type: "AWS::ApiGateway::Deployment"
DependsOn:
- "ProxyMethod"
Properties:
RestApiId: !Ref "apiGateway"
StageName: "dev"
IAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: 'sts:AssumeRole'
Policies:
- PolicyName: Logging
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: 'arn:aws:logs:*:*:*'
- PolicyName: AccessToDynamoDB
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'dynamodb:CreateTable'
- 'dynamodb:DeleteItem'
- 'dynamodb:DeleteTable'
- 'dynamodb:GetItem'
- 'dynamodb:GetRecords'
- 'dynamodb:UpdateItem'
- 'dynamodb:UpdateTable'
- 'dynamodb:PutItem'
- 'dynamodb:UpdateTable'
Resource: 'arn:aws:dynamodb:*:*:*'
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: {Ref: S3Bucket}
S3Key: {Ref: S3Key}
S3ObjectVersion: {Ref: S3ObjectVersion}
Handler: main.lambda_handler
MemorySize: 128
Role: {'Fn::GetAtt': [IAMRole, Arn]}
Runtime: python3.6
Timeout: 300
LambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt
- LambdaFunction
- Arn
Action: 'lambda:InvokeFunction'
Principal: apigateway.amazonaws.com
SourceArn: !Sub arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/*
Outputs:
apiGatewayInvokeURL:
Value: !Sub "https://${apiGateway}.execute-api.${AWS::Region}.amazonaws.com/${apiGateway}"
lambdaArn:
Value: !GetAtt "LambdaFunction.Arn"
The above Yaml file will create the Lambda function and will deploy the API, but it will show bellow error when I am trying to test the API:
{"message": "Internal server error"}
Can you please guide me what is wrong and how I can solve the problem?
The issue is related to you IntegrationHttpMethod setting. Although your APIGateway method is ANY, the IntegrationHttpMethod must always be POST for AWS Lambda.
This would lead to the following method declaration.
ProxyMethod:
Type: 'AWS::ApiGateway::Method'
Properties:
HttpMethod: ANY
ResourceId: !Ref Resource
RestApiId: !Ref apiGateway
AuthorizationType: NONE
RequestParameters:
method.request.path.proxy: true
Integration:
CacheKeyParameters:
- 'method.request.path.proxy'
RequestParameters:
integration.request.path.proxy: 'method.request.path.proxy'
Type: AWS_PROXY
IntegrationHttpMethod: POST
Uri: !Sub
- arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Arn}/invocations
- Arn:
Fn::GetAtt:
- LambdaFunction
- Arn
PassthroughBehavior: WHEN_NO_MATCH
IntegrationResponses:
- StatusCode: 200