Export RDS snapshot to s3 Using Cloudformation - amazon-web-services

I want to export RDS snapshot to s3 using cloudformation which will trigger lambda function. But i'm kind of new to this. Can someone please help me out. I've already written the script for Lambda and SNS but i'm confused that how can I do export RDS snapshot to S3 using cloudformation. I'm attaching my file of Triggering Lambda from SNS. Help will be appreciated. Thanks.
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: >
example-lambda-sns
Example CloudFormation template to subscribe a lambda to an SNS Topic.
Resources:
ExampleTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: !Sub 'example-sns-topic'
TopicName: !Sub 'example-sns-topic'
Subscription:
- Protocol: lambda
Endpoint: !GetAtt ExampleFunction.Arn
ExampleFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: mylambdafunction
Description: mylambdafunction
Handler: index.handler
Runtime: nodejs14.x
Role: arn:aws:iam::48
CodeUri: 's3://testing/tes.zip'
ExampleFunctionInvokePermission:
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref ExampleFunction
Principal: sns.amazonaws.com
ExampleTopicPolicy:
Type: 'AWS::SNS::TopicPolicy'
Properties:
Topics:
- !Ref ExampleTopic
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: 'sns:Publish'
Resource: !Ref ExampleTopic
Principal:
AWS: '*'
Condition:
ArnLike:
AWS:SourceArn: !Sub 'arn:aws:*:*:${AWS::AccountId}:*'```

following is sample example for lambda function. which will get trigger from sns. SNS will receive notification when snapshot is created.
import json
import boto3
import uuid
import datetime
import os
def lambda_handler(event, context):
snapshotarn = event['Records'][0]['Sns']['Message']
snapshotarn=snapshotarn.split()
snapshotarn=snapshotarn[10].replace('.','')
print(snapshotarn)
rds=boto3.client('rds')
export=rds.start_export_task(
ExportTaskIdentifier='export'+'-'+uuid.uuid4().hex,
SourceArn=snapshotarn,
S3BucketName=os.environ.get('S3_BUCKET_NAME'),
IamRoleArn=os.environ.get('IAM_ROLE_FOR_EXPORT_TASK'),
KmsKeyId=os.environ.get('KMS_KEY_ID'),
S3Prefix=os.environ.get('S3_PREFIX'),
)
status={
'ExportTaskIdentifier':export['ExportTaskIdentifier'],
'S3Bucket':export['S3Bucket'],
'S3Prefix':export['S3Prefix'],
'Status':export['Status'],
'ResponseMetadata':export['ResponseMetadata'],
}
print(status)
# TODO implement
return {
'statusCode': 200,
'body': status
}

Related

SQS rule with an event pattern in EventBridge not triggered [duplicate]

I am trying to set up a demo environment to try out SQS as an AWS Event Bridge Source. I tried uploading few documents to SQS to see if Event Bridge detects any change, but I don't see any events triggered. How can I test SQS as a source with AWS Event Bridge?
Resources:
Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ${AWS::StackName}
LambdaHandlerExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
EventConsumerFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.lambda_handler
Role: !GetAtt LambdaHandlerExecutionRole.Arn
Code:
ZipFile: |
import json
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
Runtime: python3.7
Timeout: 50
EventRule:
Type: AWS::Events::Rule
Properties:
Description: eventEventRule
State: ENABLED
EventPattern:
source:
- aws.sqs
resources:
- !GetAtt Queue.Arn
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: EventConsumerFunctionTarget
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref EventConsumerFunction
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventRule.Arn
SQS data events (publishing new message) are not source events for Event Bridge (EB). Only management events can be picked up by EB, e.g.:
purging of the queue
creating of new queue
deletion of a queue
Also your event rule should be more generic for that:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: eventEventRule
State: ENABLED
EventPattern:
source:
- aws.sqs
# resources:
# - !GetAtt Queue.Arn
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: EventConsumerFunctionTarget
You can also enable CloudWatch trial and detect API events for the SQS. This should enable fetching more events.
I might be late but this can benefit someone else,
have a look at this:
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-ecs-patterns.QueueProcessingFargateService.html
This will handle scaling of the Fargate container based on a number of messages in the SQS Queue.
a simplest stack can be defined using AWS CDK as following:
queue = sqs.Queue(stack, "Queue")
cluster = aws_ecs.Cluster(
stack, 'FargateCluster'
)
queue_processing_fargate_service = QueueProcessingFargateService(stack, "Service",
cluster=cluster,
memory_limit_mi_b=512,
image=ecs.ContainerImage.from_registry("test"),
command=["-c", "4", "amazon.com"],
enable_logging=False,
desired_task_count=2,
environment={
"TEST_ENVIRONMENT_VARIABLE1": "test environment variable 1 value",
"TEST_ENVIRONMENT_VARIABLE2": "test environment variable 2 value"
},
queue=queue,
max_scaling_capacity=5,
container_name="test"
)

Security Hub with Chatbot not receiving response from Custom Resource

I'm following this tutorial on enabling aws security hub with aws chatbot. I seem to be having an issue with deploying the custom resource named AddCustomActionSHResource1 in the following code, which is linked from the tutorial to here.
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys CustomAction in SecurityHub to enable sending findings to Slack via AWS Chatbot
#==================================================
# Parameters
#==================================================
Parameters:
SlackWorkSpaceID:
Description: Slack workspace ID (Copy and Paste from AWS Chatbot Configured Clients Interface)
Type: String
MinLength: 9
MaxLength: 15
AllowedPattern: ^[a-zA-Z0-9_]*$
ConstraintDescription: |
Malformed Input Parameter: Environment must contain only upper and numbers. Length should be minimum of 9 characters and a maximum of 15 characters.
SlackChannelID:
Description: Slack Channel ID
Type: String
MinLength: 9
MaxLength: 15
AllowedPattern: ^[a-zA-Z0-9_]*$
ConstraintDescription: |
Malformed Input Parameter: Environment must contain only upper and numbers. Length should be a minimum of 9 characters and a maximum of 15 characters.
# CustomActionName:
# Description: Name of the Custom Action in SecurityHub
# Type: String
# AllowedPattern: ^[a-zA-Z0-9_]*$
# Default: Send_To_Slack
# #Default: 'Send To !Sub "${ChatApplication}"'
#==================================================
# Resources
#==================================================
Resources:
#======================================================
# Lambda Role to create Custom Action
#======================================================
LambdaIAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSLambdaExecute
Path: /
#======================================================
# Lambda Policy to create Custom Action
#======================================================
LambdaIAMPolicy:
Type: AWS::IAM::Policy
DependsOn: LambdaIAMRole
Properties:
PolicyName: LambdaCreateCustomActionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'securityhub:CreateActionTarget'
Resource: '*'
Roles:
- !Ref LambdaIAMRole
#======================================================
# Lambda Function to create Custom Action
#======================================================
LambdaCreateCustomAction:
Type: AWS::Lambda::Function
DependsOn: LambdaIAMPolicy
Properties:
FunctionName: addcustomactionsecurityhub
Description: CreateCustom Action in SecurityHub
Runtime: python3.7
Handler: index.lambda_handler
Code:
ZipFile: |
import boto3
import cfnresponse
def lambda_handler(event, context):
securityhub = boto3.client('securityhub')
response = securityhub.create_action_target(Name="Send_To_Slack",Description='Send Messages to ChatApplication via AWS ChatBot',Id='SendToSlack')
responseData = {}
responseData['Data'] = response
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")
MemorySize: 128
Timeout: 10
Role: !GetAtt LambdaIAMRole.Arn
#======================================================
# Custom Resource to Invoke the Lambda Function
#======================================================
AddCustomActionSHResource1:
Type: Custom::AddCustomActionSH
DependsOn: LambdaCreateCustomAction
Properties:
ServiceToken: !GetAtt LambdaCreateCustomAction.Arn
#======================================================
# SNS Topic
#======================================================
SNSTopicAWSChatBot:
Type: AWS::SNS::Topic
Properties:
DisplayName: AWS Chatbot SNS Topic
EventTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: topicPolicyCustomaction
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sns:Publish'
Resource: '*'
Topics:
- !Ref SNSTopicAWSChatBot
#======================================================
# CloudWatch Event Rule
#======================================================
EventRuleCustomAction:
Type: AWS::Events::Rule
Properties:
Description: "SecurityHub Chatbot CustomAction"
EventPattern:
source:
- "aws.securityhub"
detail-type:
- "Security Hub Findings - Custom Action"
resources:
- !Sub 'arn:aws:securityhub:${AWS::Region}:${AWS::AccountId}:action/custom/SendToSlack'
State: "ENABLED"
Targets:
-
Arn:
Ref: "SNSTopicAWSChatBot"
Id: "OpsTopic"
ChatBotManageIAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "sns.amazonaws.com"
Action:
- "sts:AssumeRole"
SlackChannelConfig:
Type: AWS::Chatbot::SlackChannelConfiguration
Properties:
ConfigurationName: securityhubnotification
IamRoleArn: !GetAtt ChatBotManageIAMRole.Arn
LoggingLevel: NONE
SlackChannelId: !Ref SlackChannelID
SlackWorkspaceId: !Ref SlackWorkSpaceID
SnsTopicArns:
- !Ref SNSTopicAWSChatBot
When deploying this cloudformation file to cloudformation in AWS, everything deploys successfully except for AddCustomActionSHResource1. The error message is:
CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [18sa90d1-49s1-4as7-9fsc-b79ssd6csd9]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
I found some information on updating the lambda function by just adding comments to it, but I don't believe that's the issue. Most of the useful information I found was from this aws link.

SNS message is not sent to dead letter queue (DLQ) after retry delivery is exhausted

I have a subscription to a SNS topic that is configured to move a message to a DLQ if it can't be delivered successfully to a lambda function.
As described by this document, there are client-side and server-side errors. If a client-side error occurred the message is correctly moved to the DLQ but in case a server-side error occurred, the message is not moved to the DLQ. This document describes the delivery retries and the subscription does use the default delivery policy defined by the SNS topic. The retries do happen but after the retries are exhausted the message is not moved to the DLQ.
Now I wonder why the message is not moved correctly to the DLQ on server-side errors. Is there some more configuration missing?
I created the resources with the following AWS SAM template:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
lambda-test
Globals:
Function:
Timeout: 30
Resources:
EmailFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: EmailFunction
Handler: de.domain.email.App::handleRequest
Runtime: java11
Architectures:
- x86_64
MemorySize: 512
Environment:
Variables:
# https://aws.amazon.com/blogs/compute/optimizing-aws-lambda-function-performance-for-java/
JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1
EmailsTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: 'test-emails'
EmailFunctionInvokePermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref EmailFunction
Principal: sns.amazonaws.com
EmailDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: !Join ['', [!GetAtt EmailsTopic.TopicName, '-dlq']]
# Policy for DLQ: https://docs.aws.amazon.com/sns/latest/dg/sns-configure-dead-letter-queue.html
EmailDLQPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref EmailDLQ
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action:
- 'sqs:GetQueueUrl'
- 'sqs:GetQueueAttributes'
- 'sqs:SetQueueAttributes'
- 'sqs:SendMessage'
- 'sqs:ReceiveMessage'
- 'sqs:DeleteMessage'
- 'sqs:PurgeQueue'
Resource:
- !GetAtt EmailDLQ.Arn
EmailsSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref EmailsTopic
Protocol: lambda
Endpoint: !GetAtt EmailFunction.Arn
RedrivePolicy:
deadLetterTargetArn: !GetAtt EmailDLQ.Arn
And the Java function just looks like this (and throws an exception when the message body is reject):
package de.domain.email;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.SNSEvent;
public class App implements RequestHandler<SNSEvent, Object> {
public Object handleRequest(final SNSEvent input, final Context context) {
input.getRecords().forEach(r -> {
context.getLogger().log(r.getSNS().getMessage() + "\n");
if (r.getSNS().getMessage().equals("reject"))
throw new IllegalStateException("reject");
});
return null;
}
}

AWS IoT 1 Click Project SAM - 400 InvalidRequestException error

Hi I am using AWS SAM to deploy a serverless application that also relies on 1Click devices I have the template deploying fine but I have to exclude this value:
CallbackOverrides:
onClickCallback:
here is the template file
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: securitybutton
Globals:
Function:
Timeout: 10
Resources:
SecurityButtonFunction:
Type: AWS::Serverless::Function
Policies:
- AmazonSNSFullAccess
Properties:
CodeUri: src
Handler: index.lambdaHandler,
Runtime: nodejs12.x
MemorySize: 512
Events:
IoTRule:
Type: IoTRule
Properties:
Sql: SELECT * FROM 'iotbutton/*'
#TOPICS
BRSecurityButtonTopic:
Type: AWS::SNS::Topic
#IoT
#project
BRSecurityButtonProject:
Type: "AWS::IoT1Click::Project"
Properties:
ProjectName: "BRSecurityButton"
Description: "BRSecurityButton"
PlacementTemplate:
DefaultAttributes:
SNSTopic: Changeme
Location: Changeme
Theatre: Changeme
DeviceTemplates:
SecuityButtonTemplate:
DeviceType: "button"
CallbackOverrides:
onClickCallback: !GetAtt
- SecurityButtonFunction
- Arn
Outputs:
SecurityButtonFunction:
Description: "Security Button Lambda Function ARN"
Value: !GetAtt SecurityButtonFunction.Arn
Here is the resulting cloudFormation error:
Lambda function
arn:aws:lambda:us-east-1:LALALALALALA:function:securitybutton-prod-SecurityButtonFunction-6OB47JEIU192
cannot be invoked. (Service: AWSIoT1ClickProjects; Status Code: 400;
Error Code: InvalidRequestException; Request ID:
f0e94cd5-0310-4dcf-8d4a-a5ee8b102590; Proxy: null)
on the logical ID of BRSecurityButtonProject
any help would be appreciated
I found that the issue was related to IAM. Added this to the SAM template, which granted access for any IoT 1-click projects in the account to invoke the Lambda function.
SecurityButtonFunctionIoTPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt SecurityButtonFunction.Arn
Action: lambda:InvokeFunction
Principal: iot1click.amazonaws.com
SourceAccount: !Ref "AWS::AccountId"

Run AWS CLI KMS encrypt commands from Cloudformation

I have 2 cloudformation templates - one that creates a kms key and the other template uses the kms key to encrypt a env variable used in the lambda function.
I wanted to know if there is a way to run the kms encrypt command from within the cloudformation as a prior step and then use the encrypted text for the environment variable while creating the lambda function.
aws kms encrypt --key-id <key-id-output-from-stack1> --plaintext fileb://file.txt --query CiphertextBlob --output text > fileoutput.txt
This command outputs the encrypted text and I would need to use this text in the lambda function for one of the environment variables as below.
GTMLambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://test.google.com/lambdas/09yu567943879
Handler: src/lambda.handler
FunctionName: !Ref GTMLambdaFunctionName
Runtime: nodejs10.x
MemorySize: !Ref GTMLambdaMemorySize
Timeout: !Ref GTMLambdaTimeout
AutoPublishAlias: prod
Role: !GetAtt GTMLambdaRole.Arn
KmsKeyArn: !ImportValue GTMKMSKeyArn
Environment:
Variables:
url: >-
**{insert encrypted text}**
tbl_prefix: gtm-
If this is not possible is there any recommendations on how to achieve this? Thanks in advance.
You can use a custom resource for this. It will execute a Lambda function that will encrypt and return the value. That value can then be used in the environment variable.
Something like the following. Make sure to have a resource/parameter/output called KeyId with the KMS key id.
Resources:
EncryptEnvRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: DescribeImages
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action: kms:Encrypt
Effect: Allow
Resource: "*"
EncryptEnvFunction:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.6
Handler: index.handler
Role: !Sub ${EncryptEnvRole.Arn}
Timeout: 60
Code:
ZipFile:
Fn::Sub: |
import base64
import boto3
import cfnresponse
import traceback
def handler(event, context):
try:
t = event['ResourceProperties']['Value']
k = event['ResourceProperties']['KeyId']
v = base64.b64encode(boto3.client('kms').encrypt(KeyId=k, Plaintext=t.encode('utf-8'))['CiphertextBlob']).decode('utf-8')
cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, v)
except:
traceback.print_last()
cfnresponse.send(event, context, cfnresponse.FAIL, {}, 'ok')
EncryptedEnv:
Type: Custom::EncryptEnv
Properties:
ServiceToken: !Sub ${EncryptEnvFunction.Arn}
Value: "hello world"
KeyId: !ImportValue KeyId
GTMLambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://test.google.com/lambdas/09yu567943879
Handler: src/lambda.handler
FunctionName: !Ref GTMLambdaFunctionName
Runtime: nodejs10.x
MemorySize: !Ref GTMLambdaMemorySize
Timeout: !Ref GTMLambdaTimeout
AutoPublishAlias: prod
Role: !GetAtt GTMLambdaRole.Arn
KmsKeyArn: !ImportValue GTMKMSKeyArn
Environment:
Variables:
url: !Ref EncryptedEnv
tbl_prefix: gtm-