Invoke Lambda function by SNS message on local serverless-offline environment - amazon-web-services

I'm using Serverless Framework & serverless-offline plugin to develop serverless web application locally, and trying to test the following procedure.
User pushes a button, which will call API
API will invoke Lambda function and it will publish a message to SNS topic
Several lambda functions subscribing the SNS topic will be invoked
serverless.yml
plugins:
- serverless-offline
- serverless-offline-sns
functions:
publisher:
handler: publisher.main
events:
- http:
path: publish
method: post
cors: true
authorizer: aws_iam
subscriber:
handler: subscriber.main
events:
- sns: test-topic
I tested it on AWS and it worked, but I don't know how to test it locally.
serverless-offline-sns does not support subscription by lambda for now.
serverless-offline-sns supports http, https, and sqs subscriptions. email, email-json, sms, application, and lambda protocols are not supported at this time.
https://www.npmjs.com/package/serverless-offline-sns
I think this is a very common use case for serverless & event-driven architecture. How do you test this on local environment?

I was able to simulate this offline recently using the following code/config
serverless.yml
functions:
########## SNS SUBSCRIPTIONS ##########
newUser:
memorySize: 128
timeout: 120
handler: src/sns-subscribers/newUser.handler
name: sns-newUser-dev
events:
- sns:
arn: arn:aws:sns:ap-southeast-2:13XXXXXXXXXX:new-user-dev
plugins:
- serverless-offline-sns
- serverless-offline
custom:
serverless-offline-sns:
port: 4002 # a free port for the sns server to run on
debug: true
# host: 0.0.0.0 # Optional, defaults to 127.0.0.1 if not provided to serverless-offline
# sns-endpoint: http://127.0.0.1:4002 # Optional. Only if you want to use a custom endpoint
accountId: 13XXXXXXXXXX # Optional
Here's the code that triggers my offline lambda
trigger.js
const AWS = require('aws-sdk');
const sns = new AWS.SNS({
endpoint: 'http://127.0.0.1:4002',
region: 'ap-southeast-2',
});
sns.publish(
{
Message: 'new user!',
MessageStructure: 'json',
TopicArn: `arn:aws:sns:ap-southeast-2:13XXXXXXXXXX:new-user-dev`,
},
() => console.log('new user published'),
);
Run the trigger normally
node trigger.js
Note:
In your example, the way you declared the sns subscription is not yet supported AFAIK.
events:
- sns: test-topic # try using ARN and sending this to the next line
You can check this github issue for more info and updates.

Related

AWS Lambda Functions and AWS API Gateway(custom domain name) path redundancy/conflict

I am trying to remove the redundant path which is used in both my serverless configuration and aws api gateway mapping.
Problem:
Login serverless yaml
serverless.yml
frameworkversion: '>1.8'
service: ${stage}-login
provider:
name: aws
runtime: nodejs10.x
timeout: 12
functions:
login:
name: login
handler: login.handler
events:
- http:
path: login
cors: true
integration: lambda
request:
passThrough: WHEN_NO_MATCH
template:
application/json:
<response omitted>
plugins:
- serverless-offline
API mapping to my custom domain
API - login-dev
Stage - dev
Path(optional) - login
Goal:
Lambda Functions :
login - {base url}/dev/login
register - {base url}/dev/register
What happened:
login {base url}/dev/login/login
register - {base url}/dev/register/register
Actions taken:
Tried to remove the Path(optional) but it would not allow me to add another lambda function if path is omitted.
Tried to proxy(unsure if this works the way i understand it) but it doesn;t allow because an error shows that {login} is used in one of my lambda function parameters.
Removed path in serverless yaml configuration file and replaced it with blank or / - but not an option for me because i need to keep the existing configuration.
Any help is very much appreciated.
Have you tried this:
functions:
login:
name: login
handler: login.handler
events:
- http:
path: /login
................
By adding a "/" in the starting of path.

How to use AWS SAM CLI Local HttpAPI with JWT Bearer token Auth offline?

I would like to use AWS SAM JWT HttpApi Auth offline
Based on this AWS example, I decided to create the following YAML file.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs10.x
Events:
ExplicitApi: # warning: creates a public endpoint
Type: HttpApi
Properties:
ApiId: !Ref HttpApi
Method: GET
Path: /path
TimeoutInMillis: 15000
PayloadFormatVersion: "2.0"
RouteSettings:
ThrottlingBurstLimit: 600
HttpApi:
Type: AWS::Serverless::HttpApi
Properties:
FailOnWarnings: True
Auth:
Authorizers:
MyOauthAuthorizer:
IdentitySource: $request.header.Authorization
JwtConfiguration:
audience:
- audience
issuer: issuer-url
DefaultAuthorizer: MyOauthAuthorizer
Using AWS::Serverless:HttpApi based on docs creates an Amazon API Gateway HTTP API which supports JWT based auth.
I start it with
sam local start-api
However, when I query it with Postman, with or without JWT Bearer token, the request succeeds.
And the AWS query does not contain a single authenticated user object.
Running it with Debug mode does not provide any useful additional information either.
let response;
exports.lambdaHandler = async (event, context) => {
try {
// const ret = await axios(url);
response = {
statusCode: 200,
body: JSON.stringify({
message: "hello world",
event,
context,
// location: ret.data.trim()
}),
};
} catch (err) {
console.log(err);
return err;
}
return response;
};
My expectation would be that AWS SAM CLI would convert the Bearer token based on the correctly provided Issuer URL into an identity value which I can use in later operations.
Does AWS SAM Local not support this while running locally?
SAM Local unfortunately doesn't support Authorizers. There is a feature request on AWS SAM's GitHub repository to add this feature, see https://github.com/aws/aws-sam-cli/issues/137

Serverless Lambda function runs into a timeout when running Nuxt

I just managed to deploy my Nuxt application via Serverless on AWS. Basically everything works as expected but in some cases the Lambda function just runs into a timeout and can't serve my Nuxt application. Since my application is a SPA the timeout only happens during a refresh of the browser window or when I visit my page in a new tab, but only sometimes. I already increased the Lambda timeout to 30s (meets the timeout of the API Gateway) which should be enough but the timeout still occurs.
Here's my serverless.yml:
service:
name: test-app
plugins:
- serverless-nuxt-plugin
- serverless-dotenv-plugin
- serverless-domain-manager
resources:
Resources:
AssetsBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.nuxt.bucketName}
CorsConfiguration:
CorsRules:
- AllowedMethods:
- GET
- HEAD
AllowedOrigins:
- "*"
provider:
name: aws
region: eu-central-1 # this field is used for the assets files s3 path.
stage: ${env:APP_ENV}
runtime: nodejs12.x
environment:
NODE_ENV: ${env:APP_ENV}
tags: # Optional service wide function tags
usecase: test-app
environment: ${self:provider.stage}
domain: ${env:DEPLOY_DOMAIN}
custom:
nuxt:
version: app-${self:provider.stage}-v1
bucketName: test-app-static-${self:provider.stage}
cdnPath: https://cdn.XXX.com
customDomain:
domainName: ${env:DEPLOY_DOMAIN}
certificateName: ${'*.'}${env:DEPLOY_DOMAIN}
createRoute53Record: true
endpointType: 'regional'
functions:
nuxt:
handler: lambda-handler.render
memorySize: 512 # in MB with steps of 64
timeout: 30 # in seconds
events:
- http: ANY /
- http: ANY /{proxy+}
And my Lambda handler:
const awsServerlessExpress = require('aws-serverless-express');
const express = require('express');
const { Nuxt } = require('nuxt-start'); // eslint-disable-line
const nuxtConfig = require("./nuxt.config.js");
const app = express();
const nuxt = new Nuxt({
...nuxtConfig,
dev: false,
_start: true,
});
app.use(async (req, res) => {
if (nuxt.ready) {
await nuxt.ready()
}
nuxt.render(req, res)
});
const server = awsServerlessExpress.createServer(app, void 0, [
'application/javascript',
'application/json',
'application/manifest+json',
'application/octet-stream',
'application/xml',
'font/eot',
'font/opentype',
'font/otf',
'image/gif',
'image/jpeg',
'image/png',
'image/svg+xml',
'image/x-icon', // for favicon
'text/comma-separated-values',
'text/css',
'text/html',
'text/javascript',
'text/plain',
'text/text',
'text/xml',
'application/rss+xml',
'application/atom+xml',
]);
module.exports.render = (event, context) => {
awsServerlessExpress.proxy(server, event, context);
};
Additionally, I setup a CloudFront distribution in front of my API Gateway to redirect http traffic to https. So nothing really special, I guess.
Here's an example of my CloudWatch logs that shows an example timeout:
So the duration of the Lambda is pretty distributed and I can't really understand why. I even found durations of 100ms but they can get up until the timeout of 30s.
Is there anything wrong in my setup or something I missed? I'm aware of the cold start bottleneck for Lambdas but these timeout calls are not caused by a cold start.
I really appreciate your help!
I currently solved the issue by increasing the memory limit first from 512MB to 1024MB and then on a second step from 1024MB to 2048MB.
See the CloudFront diagram here (blue line)
I guess that my application is just too large with too many dependencies and modules that need to be loaded when running the Lambda. However, I'm still not sure if a memory leak or something else is causing the issue and increasing the memory limit is just hiding the issue. But if anyone has the same issue, increasing the memory seems to be a good temporary fix to at least have your application available.

Property validation failure "SNSDestination" while creating a ConfigurationSetEventDestination from CloudFormation

I'm trying to create a ConfigurationSetEventDestination using serverless resources but it not recognizing the EventDestination for value SNSDestination, here is the output.
And here the resource from serverless.yml
resources:
Resources:
HandleEmailsEvents:
Type: AWS::SNS::Topic
Properties:
DisplayName: 'Handle emails events (${self:custom.stage})'
TopicName: 'HandleEmailsEvents'
ConfigurationSet:
Type: 'AWS::SES::ConfigurationSet'
Properties:
Name: 'EMAIL_TRACKING'
ConfigurationSetEventDestination:
Type: 'AWS::SES::ConfigurationSetEventDestination'
Properties:
ConfigurationSetName: 'EMAIL_TRACKING'
EventDestination:
Name: 'EMAIL_TRACKING_DESTINATION'
Enabled: true
MatchingEventTypes:
- bounce
- complaint
SNSDestination:
TopicARN:
Ref: 'HandleEmailsEvents'
Following the documentation ConfigurationSetEventDestination EventDestination seems to not be available, but here it is with this description object.
SNSDestination is also available when creating from console
#AWS What's going on here?
Thanks,
PS: I'm not the only one...
https://forums.aws.amazon.com/thread.jspa?messageID=858616&#858616
https://forums.aws.amazon.com/thread.jspa?messageID=809004&#809004
https://forums.aws.amazon.com/thread.jspa?messageID=848013&#848013
[UPDATED]
I tried creating the same via nodejs sdk, it works, its possible, docs here.
Could be something with serverless CloudFormation generated stack?
let ses = new AWS.SES()
const destinationParams = {
ConfigurationSetName: 'test',
EventDestination: {
Name: 'xxxxx2',
MatchingEventTypes: ['send', 'reject', 'bounce', 'complaint', 'delivery', 'open', 'click'],
Enabled: true,
SNSDestination: {
TopicARN: 'arn:aws:sns:us-east-1:xxxxxxxxxx:test',
},
},
};
ses.createConfigurationSetEventDestination(destinationParams, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
I found a solution, if you are using serverless framework.
1- Create a Resource to create the ConfigurationSet in serverless.yml
resources:
Resources:
ConfigurationSet:
Type: 'AWS::SES::ConfigurationSet'
Properties:
Name: 'EMAIL_TRACKING'
2- Install serverless-ses-sns
npm install --save-dev serverless-ses-sns
3- Add to serverless.yml
plugins:
- serverless-ses-sns
4- Finally add the configuration in serverless.yml
custom:
snsDestination:
region: <region> # If absent, self:provider.region will be used
configurationSet: 'EMAIL_TRACKING'
topicArn: <topic arn> # If absent, one will be created
events: # One or more of the following
- renderingFailure
- reject
- bounce
- send
- complaint
- delivery
- open
- click
Reference for the plugin
I also stumbled upon the same issue, so as of now, SNS cannot be passed as an event destination using cloudformation. For more info refer to the link, Do checkout the NOTE, its explicitly mentioned there.

Lambda invoke Lambda via API Gateway

I can't seem to get this to work. I create 2 lambdas via C9. I'm using boto3 to invoke one lambda from another. Everything seems to work just fine via C9 but when I publish and try to access via API Gateway I keep getting "Endpoint request timed out" errors.
I know it can't be a timeout issue because I've set up my yaml files to have enough time to execute and the lambda right now are really simple (only returning a string)
here are my current yaml file. I'm wondering if maybe there are some sort of permissions I need to include for API Gateway in the second yaml
Lambda1
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: An AWS Serverless Specification template describing your function.
Resources:
api:
Type: 'AWS::Serverless::Function'
Properties:
Description: ''
Handler: api/lambda_function.lambda_handler
MemorySize: 256
Role: 'arn:aws:iam::820788395625:role/service-role/api_int-role'
Runtime: python3.6
Timeout: 30
VpcConfig:
SecurityGroupIds:
- ...
SubnetIds:
- ...
Policies: AWSLambdaFullAccess
Lambda2
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: An AWS Serverless Specification template describing your function.
Resources:
api:
Type: 'AWS::Serverless::Function'
Properties:
Description: ''
Handler: api/lambda_function.lambda_handler
MemorySize: 512
Role: 'arn:aws:iam::820788395625:role/service-role/api_int-role'
Runtime: python3.6
Timeout: 15
VpcConfig:
SecurityGroupIds:
- ...
SubnetIds:
- ...
I just set up an API Gateway endpoint directly to Lambda2 and it returned no problem. So...
API Gateway -> Lambda 2 (works)
API Gateway -> Lambda 1 -> Lambda 2 (does not work)
So for some reason when I want to call Lambda 2 via Lambda 1 over API Gateway it doesn't work.
Here is the code that is calling the 2nd Lambda
import json
import boto3
def lambda_handler(event, context):
print('call boto3 client')
lambda_client = boto3.client('lambda', region_name='us-east-1')
print('boto3 client called')
print('invoke lambda')
env_response = lambda_client.invoke(
FunctionName='cloud9-apiAlpha-api-TBSOYXLVBCLX',
InvocationType='RequestResponse',
Payload=json.dumps(event)
)
print('lambda invoked')
print('env_response')
print(env_response)
print(env_response['Payload'])
print(env_response['Payload'].read())
return {
'statusCode': 200,
'headers': {
'Content-Type': 'application/json',
'Access-Control-Allow-Methods': 'POST,GET,OPTIONS,PUT,DELETE',
'Access-Control-Allow-Origin': '*'
},
'body': 'HELLO WORLD!',
'isBase64Encoded': False
}
Now when I look at the logs it gets to print('invoke lambda') but then stops and timesout
1.Invoking a Lambda from another Lambda can't be done without some configuration. In your .yml file, permission must be specified in order to invoke another Lambda. This can be accomplished by adding an iamRoleStatements section under the provider property
or
by add the simple policy AWSLambdaRole to the existing role attached to the lambda function_1.
provider:
name: aws
runtime: <runtime goes here> # e.g. python3.6 or nodejs6.10
iamRoleStatements:
- Effect: Allow
Action:
- lambda:InvokeFunction
Resource: "*"
or do this add/attach this policy to your existing role attached to your lambda function_1
2.Invoking lambda function_1 code attached.
global LAMBDA_CLIENT
if not LAMBDA_CLIENT:
LAMBDA_CLIENT = boto3.client('lambda')
try:
encoded_payload = json.dumps({'message': 'this is an invokcation call form lambda_1'}).encode(UTF_8)
invoke_resp = lambda_client.invoke(
FunctionName='function_2',
InvocationType='RequestResponse',
Payload=encoded_payload)
status_code = invoke_resp['StatusCode']
if status_code != 200:
LOGGER.error('error ')
paylaod = invoke_resp['Payload'].read()
resp = json.loads(payload)
print(resp)
except Exception:
IF you are using InvocationType=RequestResponse then you can return some response form function_2.
Finally found the solution. The answer to my particular problem was Lambda 1 & Lambda 2 were operating over VPC thus no internet connection. Once I removed VPC from Lambda 1 the invocation of Lambda 2 worked without any problems.
Just wanted to share in case I can save anyone else a weeks worth of debugging LOL