I am working with AWS Textract and I want to analyze a multipage document, therefore I have to use the async options, so I first used startDocumentAnalysisfunction and I got a JobId as the return, But it needs to trigger a function that I have set to trigger when the SNS topic got a message.
These are my serverless file and handler file.
provider:
name: aws
runtime: nodejs8.10
stage: dev
region: us-east-1
iamRoleStatements:
- Effect: "Allow"
Action:
- "s3:*"
Resource: { "Fn::Join": ["", ["arn:aws:s3:::${self:custom.secrets.IMAGE_BUCKET_NAME}", "/*" ] ] }
- Effect: "Allow"
Action:
- "sts:AssumeRole"
- "SNS:Publish"
- "lambda:InvokeFunction"
- "textract:DetectDocumentText"
- "textract:AnalyzeDocument"
- "textract:StartDocumentAnalysis"
- "textract:GetDocumentAnalysis"
Resource: "*"
custom:
secrets: ${file(secrets.${opt:stage, self:provider.stage}.yml)}
functions:
routes:
handler: src/functions/routes/handler.run
events:
- s3:
bucket: ${self:custom.secrets.IMAGE_BUCKET_NAME}
event: s3:ObjectCreated:*
textract:
handler: src/functions/routes/handler.detectTextAnalysis
events:
- sns: "TextractTopic"
resources:
Resources:
TextractTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: "Start Textract API Response"
TopicName: TextractResponseTopic
Handler.js
module.exports.run = async (event) => {
const uploadedBucket = event.Records[0].s3.bucket.name;
const uploadedObjetct = event.Records[0].s3.object.key;
var params = {
DocumentLocation: {
S3Object: {
Bucket: uploadedBucket,
Name: uploadedObjetct
}
},
FeatureTypes: [
"TABLES",
"FORMS"
],
NotificationChannel: {
RoleArn: 'arn:aws:iam::<accont-id>:role/qvalia-ocr-solution-dev-us-east-1-lambdaRole',
SNSTopicArn: 'arn:aws:sns:us-east-1:<accont-id>:TextractTopic'
}
};
let textractOutput = await new Promise((resolve, reject) => {
textract.startDocumentAnalysis(params, function(err, data) {
if (err) reject(err);
else resolve(data);
});
});
}
I manually published an sns message to the topic and then it is firing the textract lambda, which currently has this,
module.exports.detectTextAnalysis = async (event) => {
console.log('SNS Topic isssss Generated');
console.log(event.Records[0].Sns.Message);
};
What is the mistake that I have and why the textract startDocumentAnalysis is not publishing a message and making it trigger the lambda?
Note: I haven't use the startDocumentTextDetection before using the startTextAnalysis function, though it is not necessary to call it before this.
Make sure you have in your Trusted Relationships of the role you are using:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com",
"textract.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
The SNS Topic name must be AmazonTextract
At the end your arn should look this:
arn:aws:sns:us-east-2:111111111111:AmazonTextract
I was able got this working directly via Serverless Framework by adding a Lambda execution resource to my serverless.yml file:
resources:
Resources:
IamRoleLambdaExecution:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- textract.amazonaws.com
Action: sts:AssumeRole
And then I just used the same role generated by Serverless (for the lambda function) as the notification channel role parameter when starting the Textract document analysis:
Thanks to this this post for pointing me in the right direction!
For anyone using the CDK in TypeScript, you will need to add Lambda as a ServicePrincipal as usual to the Lambda Execution Role. Next, access the assumeRolePolicy of the execution role and call the addStatements method.
The basic execution role without any additional statement (add those later)
this.executionRole = new iam.Role(this, 'ExecutionRole', {
assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
});
Next, add Textract as an additional ServicePrincipal
this.executionRole.assumeRolePolicy?.addStatements(
new PolicyStatement({
principals: [
new ServicePrincipal('textract.amazonaws.com'),
],
actions: ['sts:AssumeRole']
})
);
Also, ensure the execution role has full permissions on the target SNS topic (note the topic is created already and accessed via fromTopicArn method)
const stmtSNSOps = new PolicyStatement({
effect: iam.Effect.ALLOW,
actions: [
"SNS:*"
],
resources: [
this.textractJobStatusTopic.topicArn
]
});
Add the policy statement to a global policy (within the active stack)
this.standardPolicy = new iam.Policy(this, 'Policy', {
statements: [
...
stmtSNSOps,
...
]
});
Finally, attach the policy to the execution role
this.executionRole.attachInlinePolicy(this.standardPolicy);
If you have your bucket encrypted you should grant kms permissions, otherwise it won't work
Related
I have cdk script which makes one S3 bucket and lambda then add s3 trigger to lambda
const up_bk = new s3.Bucket(this, 'cdk-st-in-bk', { // image-resize用のbucket
bucketName: `cdk-st-${targetEnv}-resource-in-bk`,
removalPolicy: RemovalPolicy.DESTROY,
autoDeleteObjects: true,
cors: [{
allowedMethods: [
s3.HttpMethods.GET,
s3.HttpMethods.POST,
s3.HttpMethods.PUT,
s3.HttpMethods.DELETE,
s3.HttpMethods.HEAD,
],
allowedHeaders: ["*"],
allowedOrigins: ["*"],
exposedHeaders: ["ETag"],
maxAge: 3000
}]
});
const resizerLambda = new lambda.DockerImageFunction(this, "ResizerLambda", {
code: lambda.DockerImageCode.fromImageAsset("resizer-sam/resizer"),
});
resizerLambda.addEventSource(new S3EventSource(up_bk, {
events: [ s3.EventType.OBJECT_CREATED ],
}));
Now,It makes role automatically st-dev-base-stack-ResizerLambdaServiceRoleAE27CE82-1LWJL0D35A0GW
But it has only AWSLambdaBasicExecutionRole
So,when I try to access S3 from bucket there comes error like `
For example,
obj = s3_client.get_object(Bucket=bucket_name, Key=obj_key)
"An error occurred (AccessDenied) when calling the GetObject operation: Access Denied"
I guess I should add the AmazonS3FullAccess to this role.
However how can I do this??
You need to give the Lambda function permission to read from the bucket:
up_bk.grantRead(resizerLambda);
If you also need it to write to the bucket, do:
up_bk.grantReadWrite(resizerLambda);
I created a CustomResource to call a lambda function when the CloudFormation stack is created. It fails with the following error:
Received response status [FAILED] from custom resource. Message returned: User: arn:aws:sts::<account>:assumed-role/stack-role is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-east-1:<account>:function:<lambda> because no identity-based policy allows the lambda:InvokeFunction action
This is the code in the CDK:
import * as cr from '#aws-cdk/custom-resources';
const callLambda = new cr.AwsCustomResource(this, 'MyCustomResource', {
onCreate: {
service: 'Lambda',
action: 'invoke',
region: 'us-east-1',
physicalResourceId: cr.PhysicalResourceId.of(Date.now.toString()),
parameters: {
FunctionName: `my-function`,
Payload: '{}'
},
},
policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
})
});
How can I grant permissions to the stack's assumed role so that it can perform lambda:InvokeFunction?
I solved the issue by creating a role that assumes the lambda service principal, and adding a policy statement allowing the lambda:InvokeFunction.
import * as cr from '#aws-cdk/custom-resources';
import * as iam from "#aws-cdk/aws-iam";
let role = new iam.Role(this, `my-role`, {
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
});
role.addToPolicy(new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: ['lambda:InvokeFunction'],
resources: ['*']
}));
const callLambda = new cr.AwsCustomResource(this, 'MyCustomResource', {
onCreate: {
service: 'Lambda',
action: 'invoke',
region: 'us-east-1',
physicalResourceId: cr.PhysicalResourceId.of(Date.now.toString()),
parameters: {
FunctionName: `my-function`,
Payload: '{}'
},
},
policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
}),
role: role as any
});
I find fromStatements works...must be some issues with fromSdkCalls
new cr.AwsCustomResource(this, 'MyCustomResource', {
onCreate: {
service: 'Lambda',
action: 'invoke',
region: 'us-east-1',
physicalResourceId: cr.PhysicalResourceId.of(Date.now.toString()),
parameters: {
FunctionName: `my-function`,
Payload: '{}'
},
},
policy: cr.AwsCustomResourcePolicy.fromStatements([
new PolicyStatement({
effect: Effect.ALLOW,
actions: ["lambda:InvokeFunction"],
resources: ["*"],
}),
])
});
Add a ResourcePolicy to your construct.
// infer the required permissions; fine-grained controls also available
policy: AwsCustomResourcePolicy.fromSdkCalls({resources: AwsCustomResourcePolicy.ANY_RESOURCE})
How do I specify another AWS account's event bus as the target of a CloudWatch Rule using CloudFormation or CDK?
Here is an example Rule using CDK where I try to send CodeDeploy events to another account:
Rule codedeployCreateDeploymentEventRule = Rule.Builder.create(this, "CodedeployCreateDeploymentEventRule")
.description("CloudWatch event rule covering CodeDeploy CreateDeployment notifications.")
.ruleName("MyRule")
.enabled(true)
.targets(List.of(...something here...))
.eventPattern(EventPattern.builder()
.source(List.of("aws.codedeploy"))
.detail(Map.of("eventName", List.of("CreateDeployment")))
.build())
.build();
How do I specify another account's EventBus as the target? What's the syntax - is it an ARN or what?
To relay CW events from Acc1 to Acc2 in CloudFormation, three things are needed:
1. Acc2 - EventBusPolicy
AWS::Events::EventBusPolicy which allows Acc1 to submit events. Eg:
MyEventBusPolicy:
Type: AWS::Events::EventBusPolicy
Properties:
Action: events:PutEvents
EventBusName: default
Principal: 2234322123 # Account1 Id
StatementId: AcceptEventsFromAcc1
2. Acc1 - Iam Role for CW
IAM role that allows CW Events in Acc1 to publish events to Acc 2. Example:
MyCWEventsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: [events.amazonaws.com]
Action: ["sts:AssumeRole"]
Description: Role for CW event to be able to publish events to acc2
Policies:
- PolicyName: MyEventPolicy
PolicyDocument: !Sub |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"events:PutEvents"
],
"Resource": [
"arn:aws:events:${RegionId}:${AccountId}:event-bus/default"
]
}
]
}
where AccountId and RegionId are Acc2 values, not Acc1.
3. Acc1 - CW Event rule to rely events to Acc2's bus
It will use IAM role from step 2. For example, to rely CodeCommits events (I set it up before, so I know it works):
MyRule:
Type: AWS::Events::Rule
Properties:
Description: Monitor master branch of our repo and rely to Acc2
EventPattern: !Sub |
{
"source": [
"aws.codecommit"
],
"detail-type": [
"CodeCommit Repository State Change"
],
"resources": [
"${GitRepoArn}"
],
"detail": {
"event": [
"referenceCreated",
"referenceUpdated"
],
"referenceType": [
"branch"
],
"referenceName": [
"master"
]
}
}
State: ENABLED
Targets:
- Arn: !Sub "arn:aws:events:${RegionId}:${AccountId}:event-bus/default"
Id: MyEventToAcc2
RoleArn: !GetAtt MyCWEventsRole.Arn
where AccountId and RegionId are Acc2 values, not Acc1.
I am trying to log some information to a log stream I created in aws watchlog using a lambda function with aws-sdk, but I cannot get any logs even when the lambda is triggered.
This is my code,
Triggering Lambda Code
...
const lambda = new aws.Lambda();
lambda.invoke({
FunctionName: 'email-api-dev-logError',
Payload: JSON.stringify(err)
}, (err, data) => {
if(err) console.log('Lambda error is ', err);
else console.log('Lambda Data is ', data);
})
...
Lambda function
module.exports.logError = async (event) => {
const cloudwatchlogs = new aws.CloudWatchLogs();
const logEventParams = {
logEvents: [
{
message: event,
timestamp: new Date().getTime()
}
],
logGroupName: 'EmailAPIErrors',
logStreamName: 'Error'
};
cloudwatchlogs.putLogEvents(logEventParams, (err, data) => {
if (err) console.log(err, err.stack);
else console.log('Log data is ', data);
});
};
serverless.yml
iamRoleStatements:
- Effect: Allow
Action:
- lambda:InvokeFunction
Resource: "*"
- Effect: "Allow"
Action:
- "sqs:SendMessage"
- "sqs:ReceiveMessage"
Resource: "arn:aws:sqs:${self:provider.region}:*:EmailQueueDev"
- Effect: "Allow"
Action:
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Resource: "arn:aws:logs:*:*:log-group:/aws/rds/*:log-stream:*"
functions:
logError:
handler: handler.logError
I am not sure what is going wrong here, pls help me to find the possible error and the fix for it.
You can't write to custom log-group from AWS lambda. The default log-group associated with lambda will be at /aws/lambda/function-name.
This is how the AWS lambda is designed. AWS Lambda service is kind of convention over configuration. So, the default log stream pattern is already defined.
You can do similar behavior if you happen to use EC2 machine by installing cloudwatch agent on machine.
cloudwatch agent configuration-EC2
I've been all over the web searching for an answer to this.
Essentially, we're spinning up an API using Swagger, which is awesome and works great, but one thing doesn't work... When we make a call to an Endpoint, we get a 500 error (it's not a 500 error that we're providing either it's one from AWS). The error states "Execution failed due to configuration error: Invalid permissions on Lambda function" (https://youtu.be/H4LM_jw5zzs <- This is a video, from another user, of the error I'm getting).
I've gone down many ratholes, and have found an answer... It involves using the AWS CLI and looks a bit like this:
aws lambda add-permission \
--function-name FUNCTION_NAME \
--statement-id STATEMENT_ID \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com \
--source-arn "arn:aws:execute-api:us-east-1:ACCOUNT_ID:API_ID/*/METHOD/ENDPOINT"
This is great and all, but we are using CloudFormation to spin up everything and we want this to be automated. Is there an easier way to go about this? Is there something in CloudFormation that will give us the resource policy that we need?
I'm hitting a bit of a wall with this, but I've been working on it for a few hours today and it's a bit of a blocker for our API release, so any help would be much appreciated. :)
There is a CloudFormation solution to this problem. See the following CloudFormation snippet:
"Permission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": { "Fn::GetAtt": [ "Lambda", "Arn" ] },
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": { "Fn::Join": [ "", [
"arn:aws:execute-api:",
{ "Ref": "AWS::Region" }, ":",
{ "Ref": "AWS::AccountId" }, ":",
{ "Ref": "API" },
"/*/*/*"
] ] }
}
}
This grants API Gateway permissions to launch your Lambda function. Variables in this snippet you need to change are Lambda (line 4) and API (line 11).
For the invoke permissions:
"APIInvokePermission": {
"Type": "AWS::Lambda::Permission",
"Properties": {
"FunctionName": {
"Ref": "YOUR_LAMBDA_FUNCTION_RESOURCE_NAME"
},
"Action": "lambda:InvokeFunction",
"Principal": "apigateway.amazonaws.com",
"SourceArn": {
"Fn::Sub": "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${YOUR_REST_API_RESOURCE_NAME}/*/*/*"
}
}
},
Thanks https://twitter.com/edjgeek for helping me get this straight in my head.
This GIST shows how to use AWS::Serverless:Function with Events to automatically generate the needed AWS::Lambda::Permission to allow APIGateway (for a given route) to invoke your Lambda:
https://gist.github.com/rainabba/68df1567cbd0c4930d428c8953dc2316
Both of the following approaches assume an api such as (many fields omitted for readability):
MyApi:
Type: 'AWS::Serverless::Api'
Properties:
DefinitionBody:
Fn::Transform:
Name: AWS::Include
Parameters:
Location: openapi.yaml
Using "SAM Events"
The most relevant bit (I've omitted many required fields):
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
Events:
MyRouteEventToProxy:
Type: Api
Properties:
Method: POST
Path: '/some-route/{pathParm}'
RestApiId: !Ref MyApi # ResourceName of AWS::Serverless::Api
Auth:
Authorizer: NONE
Using "openapi binding"
If you'd rather declare the binding in the openapi.yaml, then see the following project (no Lambda/Events required). This approach requires an explict role to allow invoke.
template.yaml relevant bits:
MyLambdaFunction:
Type: 'AWS::Serverless::Function'
Properties:
# Events: # No need for Events when binding from openapi.yaml
MyHttpApiRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
openapi.yaml relevant bits:
paths:
post:
x-amazon-apigateway-integration:
httpMethod: POST
uri:
Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambdaFunction.Arn}:live/invocations"
contentHandling: "CONVERT_TO_TEXT"
type: aws_proxy
credentials:
Fn::GetAtt: [MyHttpApiRole, Arn]
https://github.com/aws-samples/sessions-with-aws-sam/tree/master/http-api-direct-integration