Cache async API result outside of AWS Lambda execution handler - amazon-web-services

Was looking at using the SSM Parameter Store SDK to grab secrets for a lambda function. I'm also using epsagon to wrap the async handler function. Epsagon needs to be initialized with some secrets, and then is used to wrap the handler function:
import * as epsagon from 'epsagon'
epsagon.init({
token: EPSAGON_ACCOUNT_TOKEN,
})
export const lambdaHandler = epsagon.lambdaWrapper(async (event) => {
// do stuff
})
Started using aws-parameter-cache to grab config values from SSM Param Store, but since they are resolved with an API call, it takes an await to get the values fully resolved.
import { ssmParameter } from 'aws-parameter-cache'
const param = ssmParameter({ name: 'foo' })
const value = await param.value; // <-- can only be done inside an async function (nodejs12)
Since we don't yet have top level await in nodejs12, is there a way to resolve the variables outside of the handler function? Is it possible to wait for the API call for await param.value to finish so that I can initialize epsagon with a value stored in SSM Param Store?
import * as epsagon from 'epsagon'
import { ssmParameter } from 'aws-parameter-cache'
const ssmParam = ssmParameter({ name: 'epsagonToken' })
const epsagonToken = await ssmParam.value // fails since outside of async func
epsagon.init({
token: epsagonToken,
})
export const lambdaHandler = epsagon.lambdaWrapper(async (event) => {
const epsagonToken = await ssmParam.value // works here but too late
})
Would this "just work" in nodejs 14.3.0 with top-level await? Custom runtime?
Or maybe some form of never-rejecting top-level async function, like in the top answer to this: how-can-i-use-async-await-at-the-top-level?
Need the handler to be the callback to the top-level async function--from what I've read this is essentially how top-level async works in 14.3. Looking for way to store all secrets in SSM Param store and reduce cf template ENV variable mappings.

Basically there is no easy way to do top-level await in this case, but there are some easy workarounds around it. For example, here is an implementation of another wrapper that you can use to initialize Epsagon:
import * as epsagon from 'epsagon'
import { ssmParameter } from 'aws-parameter-cache'
const ssmParam = ssmParameter({ name: 'epsagonToken' })
const withEpsagon = (wrapped) => {
let epsagonInited = false
const epsagonizedFunction = epsagon.lambdaWrapper(wrapped)
return async (event, context, callback) => {
if (!epsagonInited) {
const epsagonToken = await ssmParam.value
epsagon.init({
token: epsagonToken,
})
epsagonInited = true
}
return epsagonizedFunction(event, context, callback)
}
}
export const lambdaHandler = withEpsagon(async (event) => {
// your code here
})
This code will resolve the SSM parameter on its first execution (right after a cold start, which is a time you would have to spend on the cold start anyway), and memorize that it already initialized Epsagon so you don't have to waste time on every time the Lambda is invoked.

Related

How to use one APIGateway to multiple lambda

I have two lambda functions .
Now I want to use one api for these two.
Then my code is like this
const api = new apigateway.RestApi(this, 'ServerlessRestApi', {
restApiName: `AWSCDKTest-${systemEnv}`,
cloudWatchRole: false
});
api.root.addMethod('GET', new apigateway.LambdaIntegration(helloLambda));
api.root.addMethod('GET', new apigateway.LambdaIntegration(happyLambda));
Howeber it says GET is doubled.
So I made two API
const api = new apigateway.RestApi(this, 'ServerlessRestApi_hello', {
restApiName: `AWSCDK-Viral-${systemEnv}`,
cloudWatchRole: false
});
api.root.addMethod('GET', new apigateway.LambdaIntegration(helloLambda));
const api2 = new apigateway.RestApi(this, 'ServerlessRestApi_happy', { cloudWatchRole: false });
api2.root.addMethod('GET', new apigateway.LambdaIntegration(happyLambda));
It works, but it makes two API.
What is the best practice to use one API for two lambda??
API root:
GET "https://example_api_endpoint/" invokes helloLambda then inside this lambda function call the AWS-SDK lambda API method invoke() to trigger the execution of happyLambda within the first invocation in sequence. Otherwise, you cannot have two lambda functions for a single API resource. Further reading: AWS Lambda Fanout pattern.
One lambda cannot be linked to exact same API path + Http verb. Here are some alternatives -
a) have different api paths, each calling different lambda
b) have one lambda triggered by API gateway, which triggers other lambda that you need (within it's code)
c) have the lambda integration to the endpoint that sends a message to SNS, and have multiple lambda subscribe to the SNS via SQS.
try this
this.api = new RestApi(this, 'ServerlessRestApi', {
restApiName: "myapi",
cloudWatchRole: false
});
this.api.root.addResource(resource).addMethod('GET', new LambdaIntegration(func));
try this
// ./helloService.helloLambda.js file sample
export const handler = async (event, context) => {
return {
statusCode: 200,
headers: {},
body: 'helloLambda',
};
};
// ./helloService.happyLambda.js file sample
export const handler = async (event, context) => {
return {
statusCode: 200,
headers: {},
body: 'happyLambda',
};
};
// .helloService file sample
import { Construct } from 'constructs';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
export class HelloService extends Construct {
constructor(scope: Construct, id: string) {
super(scope, id);
const helloLambdaFunction = new NodejsFunction(this, 'helloLambda');
const happyLambdaFunction = new NodejsFunction(this, 'happyLambda');
const api = new apigateway.RestApi(this, 'hello-api', {
description: 'This service is Happy.',
});
const helloLambdaPath = api.root.addResource('helloLambda');
// path name https://{createdId}.execute-api.{region}.amazonaws.com/prod/helloLambda
helloLambdaPath.addMethod('GET', new apigateway.LambdaIntegration(helloLambdaFunction));
const happyLambdaPath = api.root.addResource('happyLambda');
// path name https://{createdId}.execute-api.{region}.amazonaws.com/prod/happyLambda
happyLambdaPath.addMethod('GET', new apigateway.LambdaIntegration(happyLambdaFunction));
}
}

Processing results of Lambda calling another Lambda

I have an AWS Lambda Function A that calls another Lambda Function B
For sake of discussion I want to invoke it synchronously - wait for results
and process them.
I want to do something like this in Lambda A:
let results = lambda.invoke(LambdaB);
// Do some stuff with results
The issue is that when I use the SDK APi to invoke Lambda B, I pass in a function that processes the results of that invocation. The processing function gets invoke and processes the results, but I'm noticing that the
// Do some other stuff with results
line is executing before that processing function completes, so the results
are not available yet. I'm still somewhat new to the NodeJS way of doing things, so what is the paradigm to ensure that I have the results I want before I move on to more processing with Lambda A? Here is what I have in a nuthell:
// Lambda A code
let payLoad = undefined;
let functionName = "LambdaB";
let params = {
FunctionName: functionName,
InvocationType: "RequestResponse",
LogType: "Tail",
Payload: '{name: "Fred"}'
};
lambda.invoke(params, function(err,data) {
if (err) {
// process error
} else {
payLoad = JSON.parse(data.Payload);
// payLoad is set properly here.
}
});
console.log("Do stuff with results');
console.log("PAYLOAD=" + payLoad);
// payLoad is undefined here
// How can I ensure it is set by the time I get here
// or do I need to use a different paradigm?
You will need to use async/await.
So your code will look like that :
exports.handler = async () => {
// Lambda A code
let functionName = "LambdaB";
let result = undefined;
let payload = undefined;
let params = {
FunctionName: functionName,
InvocationType: "RequestResponse",
LogType: "Tail",
Payload: '{name: "Fred"}'
};
try {
result = await lambda.invoke(params).promise();
payload = JSON.parse(result.Payload);
} catch (err) {
console.log(err);
}
console.log("Do stuff with results');
console.log(payload);
return;
}
Watchout : the lambda handler has to be an async function to use async/await in it !

Step Function Triggered In a loop

I am starting a step function from Lambda and the Lambda function is tied to an API Gateway. For some reason, when I try to test the Lambda function, I see hundreds of executions failed and running in loop. I just triggered the step function once. I am missing something here. Can you please advise.
const AWS = require("aws-sdk");
const uuidv4 = require("uuid/v4");
/*----------------------------------------------------------------------- */
/* Implementation */
/*----------------------------------------------------------------------- */
exports.handler = async event => {
var _dt = await ExecuteStepFunction()
return _dt;
}
function ExecuteStepFunction() {
const stepFunctions = new AWS.StepFunctions();
return new Promise((res, rej) => {
var params = {
stateMachineArn: 'arn:aws:states:us-east-1:xxxxxxxxxxxxx:stateMachine:xxTestSateMachine',
input: JSON.stringify(''),
name: uuidv4()
};
stepFunctions.startExecution(params, function (err, data) {
if (err) {
rej(err);
}
else {
res(data);
}
});
});
}
I tried thIS approach provided in the this link (https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html) where the API gateway directly triggers the step function but I am receiving the following error. After trying to fix this, I move to the above option of starting the function using the API.
{
"__type": "com.amazon.coral.service#UnrecognizedClientException",
"message": "The security token included in the request is invalid"
}

AWS Lambda seems exiting before completion

I have a very simple lambda function (nodeJS) which put the event received in kinesis stream. Here is the source code:
'use strict';
const AWS = require('aws-sdk');
const kinesis = new AWS.Kinesis({apiVersion: '2013-12-02'});
exports.handler = async (event, context, callback) => {
let body = JSON.parse(event.body);
let receptionDate = new Date().toISOString();
let partitionKey = "pKey-" + Math.floor(Math.random() * 10);
// Response format needed for API Gateway
const formatResponse = (status, responseBody) => {
return {
statusCode: status,
headers: { "Content-Type": "application/json" },
body: JSON.stringify(responseBody)
}
}
// body.events is an array of events. Just add the reception date in each events.
for(let e of body.events) {
e.reception_date = receptionDate;
}
console.log("put In kinesis stream");
let kinesisParams = {
Data: new Buffer(JSON.stringify(body) + "\n"),
PartitionKey: partitionKey,
StreamName: 'event_test'
};
kinesis.putRecord(kinesisParams, (err, res) => {
console.log("Kinesis.putRecord DONE");
if(err) {
console.log("putRecord Error:", JSON.stringify(err));
callback(null, formatResponse(500, "Internal Error: " + JSON.stringify(err)));
} else {
console.log("putRecord Success:", JSON.stringify(res));
callback(null, formatResponse(200));
}
});
};
When this code is executed, here are the logs in cloudwatch:
START RequestId: 5d4d7526-1a40-401f-8417-06435f0e5408 Version: $LATEST
2019-01-11T09:39:11.925Z 5d4d7526-1a40-401f-8417-06435f0e5408 put In kinesis stream
END RequestId: 5d4d7526-1a40-401f-8417-06435f0e5408
REPORT RequestId: 5d4d7526-1a40-401f-8417-06435f0e5408 Duration: 519.65 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 28 MB
It seems that kinesis.putRecord is not called... I don't see anything in kinesis stream logs. I'm certainly wrong somewhere, but I don't know where !
kinesis.putRecord is an asynchronous operation, which calls callback (The second param) when it's finished (whether successful or with an error).
async function is a function that returns a promise. Lambda will finish its execution when this promise is resolved, even if there are other asynchronous operations which are not done yet.
Since your function returns nothing, then the promise is immediately resolved when the function ends and therefore the execution will be finished immediately - without waiting to your async kinesis.putRecord task.
When using an async handler, you don't need to call callback. Instead, you return what ever you want, or throw an error. Lambda will get it and respond respectively.
So you have 2 options here:
Since you don't have any await in your code, just remove the async. In this case Lambda is waiting for the event loop to be emtpy (Unless you explicitly change context.callbackWaitsForEmptyEventLoop)
Change the kinesis.putRecord to something like:
let result;
try {
result = await kinesis.putRecord(kinesisParams).promise();
} catch (err) {
console.log("putRecord Error:", JSON.stringify(err));
throw Error(formatResponse(500, "Internal Error: " + JSON.stringify(err));
}
console.log("putRecord Success:", JSON.stringify(result));
return formatResponse(200);
In the second option, the lambda will keep running until kinesis.putRecord is finished.
For more information about Lambda behavior in this case, you can see the the main code which execute your handler under /var/runtime/node_modules/awslambda/index.js in the lambda container.
#ttulka could you explain a bit more? Give advices or code samples ? –
Adagyo
It's about the async processing evolution in JavaScript.
First, everything was done with callback, it's the oldest approach. Using callbacks everywhere leads to "Callback Hell" (http://callbackhell.com).
Then Promises was introduced. Working with Promises looks a bit like working with Monads, everything is packed into a "box" (Promise), so you have to chain all your calls:
thisCallReturnsPromise(...)
.then(data => ...)
.then(data => ...)
.then(data => ...)
.catch(err => ...)
Which is a bit unnatural to humans, so ECMAScript 2017 proposed a syntactic sugar in async functions (async/await) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Async/await syntax allows you to work with async promises like with a normal sync code:
const data = await thisCallReturnsPromise(...)
Don't forget, the await call must be inside an async function:
async () => {
const data = await thisCallReturnsPromise(...)
return await processDataAsynchronouslyInPromise(data)
}
AWS Lambda supports Node.js v8.10, which fully implements this syntax.
Just found the solution: Removing "async" keyword make it work !
exports.handler = (event, context, callback) => { ... }

Stripe Error: No signatures found matching the expected signature for payload

I have a stripe webhook that call a Firebase function. In this function I need to verify that this request comes from Stripe servers. Here is the code :
const functions = require('firebase-functions');
const bodyParser = require('body-parser');
const stripe = require("stripe")("sk_test_****");
const endpointSecret = 'whsec_****';
const app = require('express')();
app.use(bodyParser.json({
verify: function (req, res, buf) {
var url = req.originalUrl;
if (url.startsWith('/webhook')) {
req.rawBody = buf.toString()
}
}
}));
app.post('/webhook/example', (req, res) => {
let sig = req.headers["stripe-signature"];
try {
console.log(req.bodyRaw)
let event = stripe.webhooks.constructEvent(req.body, sig, endpointSecret);
console.log(event);
res.status(200).end()
// Do something with event
}
catch (err) {
console.log(err);
res.status(400).end()
}
});
exports.app = functions.https.onRequest(app);
As mentioned in Stripe Documentation, I have to use raw body to perform this security check.
I have tried with my current code and with :
app.use(require('body-parser').raw({type: '*/*'}));
But I always get this error :
Error: No signatures found matching the expected signature for payload. Are you passing the raw request body you received from Stripe? https://github.com/stripe/stripe-node#webhook-signing
Cloud Functions automatically parses body content of known types. If you're getting JSON, then it's already parsed and available to you in req.body. You shouldn't need to add other body parsing middleware.
If you need to process the raw data, you should use req.rawBody, but I don't think you'll need to do that here.
Here is what is working for me:
add this line:
app.use('/api/subs/stripe-webhook', bodyParser.raw({type: "*/*"}))
(The first argument specifies which route we should use the raw body parser on. See the app.use() reference doc.)
just before this line:
app.use(bodyParser.json());
(it doesn't affect all your operation, just this: '/api/subs/stripe-webhook')
Note: If you are using Express 4.16+ you can replace bodyParser by express:
app.use('/api/subs/stripe-webhook', express.raw({type: "*/*"}));
app.use(express.json());
Then:
const endpointSecret = 'whsec_........'
const stripeWebhook = async (req, res) => {
const sig = req.headers['stripe-signature'];
let eventSecure = {}
try {
eventSecure = stripe.webhooks.constructEvent(req.body, sig, endpointSecret);
//console.log('eventSecure :', eventSecure);
}
catch (err) {
console.log('err.message :', err.message);
res.status(400).send(`Webhook Secure Error: ${err.message}`)
return
}
res.status(200).send({ received: true });
}
Here is code which is working for me:
app.use(bodyParser.json({
verify: function (req, res, buf) {
var url = req.originalUrl;
if (url.startsWith('/stripe')) {
req.rawBody = buf.toString();
}
}
}));
And then pass the req.rawBody for verification
stripe.checkWebHook(req.rawBody, signature);
Reference: https://github.com/stripe/stripe-node/issues/341
2 things to note:
pass req.rawBody instead of req.body to constructEvent
const event = stripe.webhooks.constructEvent(
req.rawBody,
sig,
STRIPE_WEBHOOK_SECRET
);
Make sure you're using the correct webhook secret. It's unique per webhook url!
2021 - Solution
I faced that error, and after a lot research I could not figure out the problem easily, but finally I could do it based in my architecture below:
//App.js
this.server.use((req, res, next) => {
if (req.originalUrl.startsWith('/webhook')) {
next();
} else {
express.json()(req, res, next);
}
});
//routes.js
routes.post(
'/webhook-payment-intent-update',
bodyParser.raw({ type: 'application/json' }),
//your stripe logic (Im using a controller, but wherever)
(req, res) => {
stripe.webhooks.constructEvent(...)
}
)
Two big warnings to pay attention:
Make sure to send the req.headers['stripe-signature']
Make sure that your endpointSecret is right, if not it will still saying the same error
Tips:
Test it locally by installing the Stripe CLI: https://stripe.com/docs/webhooks/test
Verify your key on stripe dashboard or you can also make sure if you have the right key by verifying you stripe log as below:
I hope it helps you. :)
// Use JSON parser for all non-webhook routes
app.use(
bodyParser.json({
verify: (req, res, buf) => {
const url = req.originalUrl;
if (url.startsWith('/api/stripe/webhook')) {
req.rawBody = buf.toString();
}
}
})
);
The above code will look fine for the above answers. But even I was made one mistake. After put the same thing I got the same error.
Finally, I've figured it out if you're configured body-parser below the rawBody code then it'll work.
Like this
// Use JSON parser for all non-webhook routes
app.use(
bodyParser.json({
verify: (req, res, buf) => {
const url = req.originalUrl;
if (url.startsWith('/api/stripe/webhook')) {
req.rawBody = buf.toString();
}
}
})
);
// Setup express response and body parser configurations
app.use(express.json());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
Hopefully, it'll help someone.
It is late but will help others
Github answer
const payload = req.body
const sig = req.headers['stripe-signature']
const payloadString = JSON.stringify(payload, null, 2);
const secret = 'webhook_secret';
const header = stripe.webhooks.generateTestHeaderString({
payload: payloadString,
secret,
});
let event;
try {
event = stripe.webhooks.constructEvent(payloadString, header, secret);
} catch (err) {
console.log(`Webhook Error: ${err.message}`)
return res.status(400).send(`Webhook Error: ${err.message}`);
}
switch (event.type) {
case 'checkout.session.completed': {
......
enter code here
If you are trying to add a stripe webhook into your NextJS API Route, here's how to do so (ref):
import initStripe from "stripe";
import { buffer } from "micro";
import { NextApiRequest, NextApiResponse } from "next";
export const config = { api: { bodyParser: false } };
const handler = async (req: NextApiRequest, res: NextApiResponse) => {
const stripe = initStripe(process.env.STRIPE_SECRET_KEY||'');
const signature = req.headers["stripe-signature"];
const signingSecret = process.env.STRIPE_WEBHOOK_SECRET || '';
const reqBuffer = await buffer(req);
let event;
try {
event = stripe.webhooks.constructEvent(reqBuffer, signature, signingSecret);
} catch (error: any) {
console.log(error);
return res.status(400).send(`Webhook error: ${error?.message}`);
}
console.log({ event });
res.send({ received: true });
};
export default handler;
This is using buffer from the micro library, in combination with the modifying the default API request to use request's rawbody. In some frameworks (like NextJs), rawBody doesn't come OOTB, hence the workaround of retrieving the rawbody by reqBuffer, which is needed in the stripe.webhooks.constructEvent event.
I was able to obtain data from one webhook but not from a second one: the problem was that the secret key I used was the same as the one used for the first webhook, but I found out that every webhook has a different key, that's way I got that same message.
AWS API Gateway + Lambda (Express.js CRUD) I'm using this for Stripe webhook endpoint and it works for me:
app.use(require('body-parser').text({ type: "*/*" }));
This happened to me when sending a test webhook from the Stripe dashboard after I had renamed a firebase cloud function. All my other functions were working fine. Solved by re-setting in the terminal
firebase functions:config:set stripe.webhook_signature="Your webhook signing secret"
(if you're using that) and redeploying the functions firebase deploy --only functions
On a second occasion I solved the problem by rolling the stripe signature in the stripe dashboard.
Please use this script
app.use(
bodyParser.json({
verify: (req, res, buf) => {
req.rawBody = buf;
},
})
);
My fave was combining two of above great answers.
Then you can use req.rawbody when you construct the event.
Replace "webhook" with whatever route you wish you have a raw body for.
app.use(
"/webhook",
express.json({
verify: (req, res, buf) => {
req.rawBody = buf.toString();
},
})
);
BEFORE
app.use(express.json());
Works well if you are using routes and controllers.
To use raw body in express with a specific endpoint in a seperated middleware, my solution is just enabling router to use express.raw for the webhook endpoint.
-node.js v12
-express.js v4.17.1
export const handleBodyRequestParsing = (router: Router): void => {
router.use('/your_webhook_endpoint', express.raw({ type: '*/*' }))
router.use(express.json({ limit: '100mb' }))
router.use(express.urlencoded({ extended: true }))
}
Here is the Quick Tip which may save your hours !
If you are adding express payment to your exciting express app sometimes you may already pass your request as json in the beginning of application by using express middleware app.use(json()); or any other middleware (Bodyparser for example).
If you are doing that then change that to omit your webhook url
Exmaple:
Assume your payment webhook url is /paments/webhhok
app.use((req, res, next) => {
if (req.originalUrl.includes("/payments/webhook")) {
next();
} else {
express.json()(req, res, next);
}
});
When using Stripe in Express, if you have the following line in your code;
app.use(express.json());
it is going to prevent you from providing the raw body to the Stripe even when you explicitly set "bodyParser.raw", which will throw an error. This was the reason my code failed. Finally sorted it out.
I tried all the solutions above and no one worked, and figured out that the only solution was not to use express at all for this endpoint. you just have to create another http function
export const webhook = functions.https.onRequest(async (req, res) => {
try {
const sig = req.headers['stripe-signature']
const endpointSecret = 'web_secret'
const event = stripe.webhooks.constructEvent(
req.rawBody,
sig,
endpointSecret
)
console.log(event.data.object)
res.status(200).send(event.data.object)
} catch (err) {
console.error('ocorreu um erro', err)
res.status(400).send(`Webhook Error: ${err.message}`)
}
})