Hi i am making a finance application that depends on Events from blockchain,
Basically i update my database based on Events i receive using web3js, and when user asks i sign with private key for the contract to be able to give user Money.
My only concern is can i depend on events? like can there be a case where i miss events?
here is my code for doing so :
const contract = new web3.eth.Contract(abi, contract_address)
const stale_keccak256 = "0x507ac39eb33610191cd8fd54286e91c5cc464c262861643be3978f5a9f18ab02";
const unStake_keccak256 = "0x4ac743692c9ced0a3f0052fb9917c0856b6b12671016afe41b649643a89b1ad5";
const getReward_keccak256 = "0x25c30c62c42b51e4f667b70ef60f1f683c376f6ace28312ed45a40665e01af37";
let userRepository: Repository<UserData> = connection.getRepository(UserData);
let globalRepository: Repository<GlobalStakingInfo> = connection.getRepository(GlobalStakingInfo);
let userStakingRepository: Repository<UserStakingInfo> = connection.getRepository(UserStakingInfo);
let transactionRepository: Repository<Transaction> = connection.getRepository(Transaction);
const topics = []
web3.eth.subscribe('logs', {
address: contract_address, topics: topics
},
function (error: Error, result: Log) {
if (error) console.log(error)
}).on("data", async function (log: Log) {
let response: Boolean = false;
try {
response = await SaveTransaction(rpc_url, log.address, log.transactionHash, transactionRepository)
} catch (e) {
}
if (response) {
try {
let global_instance: GlobalStakingInfo | null = await globalRepository.findOne({where: {id: 1}})
if (!global_instance) {
global_instance = new GlobalStakingInfo()
global_instance.id = 1;
global_instance = await globalRepository.save(global_instance);
}
if (log.topics[0] === stale_keccak256) {
await onStake(web3, log, contract, userRepository, globalRepository, userStakingRepository, global_instance);
} else if (log.topics[0] === unStake_keccak256) {
await onUnStake(web3, log, contract, userStakingRepository, userRepository, globalRepository, global_instance)
} else if (log.topics[0] === getReward_keccak256) {
await onGetReward(web3, log, userRepository)
}
} catch (e) {
console.log("I MADE A BOBO", e)
}
}
}
)
The Code works and everything, i am just concerned if i could maybe miss a event? cause finance is related and people will lose money if missing event is a thing.
Please advise
You can increase redundancy by adding more instances of the listener connected to other nodes.
And also by polling past logs - again recommended to use a separate node.
Having multiple instances doing practically the same thing will result in multiplication of incoming data, so don't forget to store only unique logs. This might be a bit tricky, because theoretically one transaction ID can produce the same log twice (e.g. through a multicall), so the unique key could be a combination of a transaction ID as well as the log index (unique per block).
We're trying to develop a self-invoking lambda to process S3 files in chunks. The lambda role has the policies needed for the invocation attached.
Here's the code for the self-invoking lambda:
export const processFileHandler: Handler = async (
event: S3CreateEvent,
context: Context,
callback: Callback,
) => {
let bucket = loGet(event, 'Records[0].s3.bucket.name');
let key = loGet(event, 'Records[0].s3.object.key');
let totalFileSize = loGet(event, 'Records[0].s3.object.size');
const lastPosition = loGet(event, 'position', 0);
const nextRange = getNextSizeRange(lastPosition, totalFileSize);
context.callbackWaitsForEmptyEventLoop = false;
let data = await loadDataFromS3ByRange(bucket, key, nextRange);
await database.connect();
log.debug(`Successfully connected to the database`);
const docs = await getParsedDocs(data, lastPosition);
log.debug(`upserting ${docs.length} records to database`);
if (docs.length) {
try {
// upserting logic
log.debug(`total documents added: ${await docs.length}`);
} catch (err) {
await recurse(nextRange.end, event, context);
log.debug(`error inserting docs: ${JSON.stringify(err)}`);
}
}
if (nextRange.end < totalFileSize) {
log.debug(`Last ${context.getRemainingTimeInMillis()} milliseconds left`);
if (context.getRemainingTimeInMillis() < 10 * 10 * 10 * 6) {
log.debug(`Less than 6000 milliseconds left`);
log.debug(`Invoking next iteration`);
await recurse(nextRange.end, event, context);
callback(null, {
message: `Lambda timed out processing file, please continue from LAST_POSITION: ${nextRange.start}`,
});
}
} else {
callback(null, { message: `Successfully completed the chunk processing task` });
}
};
Where recurse is an invocation call to the same lambda. Rest of the things work as expected it just times out whenever the call stack comes on this invocation request:
const recurse = async (position: number, event: S3CreateEvent, context: Context) => {
let newEvent = Object.assign(event, { position });
let request = {
FunctionName: context.invokedFunctionArn,
InvocationType: 'Event',
Payload: JSON.stringify(newEvent),
};
let resp = await lambda.invoke(request).promise();
console.log('Invocation complete', resp);
return resp;
};
This is the stack trace logged to CloudWatch:
{
"errorMessage": "connect ETIMEDOUT 63.32.72.196:443",
"errorType": "NetworkingError",
"stackTrace": [
"Object._errnoException (util.js:1022:11)",
"_exceptionWithHostPort (util.js:1044:20)",
"TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)"
]
}
Not a good idea to create a self-invoking lambda function. In case of an error (could also be a bad handler call on AWS side) a lambda function might re-run several times. Very hard to monitor and debug.
I would suggest using Step Functions. I believe this tutorial can help Iterating a Loop Using Lambda
From the top of my head, if you prefer not dealing with Step Functions, you could create a Lambda trigger for an SQS queue. Then you pass a message to the queue if you want to run the lambda function another time.
I have a stripe webhook that call a Firebase function. In this function I need to verify that this request comes from Stripe servers. Here is the code :
const functions = require('firebase-functions');
const bodyParser = require('body-parser');
const stripe = require("stripe")("sk_test_****");
const endpointSecret = 'whsec_****';
const app = require('express')();
app.use(bodyParser.json({
verify: function (req, res, buf) {
var url = req.originalUrl;
if (url.startsWith('/webhook')) {
req.rawBody = buf.toString()
}
}
}));
app.post('/webhook/example', (req, res) => {
let sig = req.headers["stripe-signature"];
try {
console.log(req.bodyRaw)
let event = stripe.webhooks.constructEvent(req.body, sig, endpointSecret);
console.log(event);
res.status(200).end()
// Do something with event
}
catch (err) {
console.log(err);
res.status(400).end()
}
});
exports.app = functions.https.onRequest(app);
As mentioned in Stripe Documentation, I have to use raw body to perform this security check.
I have tried with my current code and with :
app.use(require('body-parser').raw({type: '*/*'}));
But I always get this error :
Error: No signatures found matching the expected signature for payload. Are you passing the raw request body you received from Stripe? https://github.com/stripe/stripe-node#webhook-signing
Cloud Functions automatically parses body content of known types. If you're getting JSON, then it's already parsed and available to you in req.body. You shouldn't need to add other body parsing middleware.
If you need to process the raw data, you should use req.rawBody, but I don't think you'll need to do that here.
Here is what is working for me:
add this line:
app.use('/api/subs/stripe-webhook', bodyParser.raw({type: "*/*"}))
(The first argument specifies which route we should use the raw body parser on. See the app.use() reference doc.)
just before this line:
app.use(bodyParser.json());
(it doesn't affect all your operation, just this: '/api/subs/stripe-webhook')
Note: If you are using Express 4.16+ you can replace bodyParser by express:
app.use('/api/subs/stripe-webhook', express.raw({type: "*/*"}));
app.use(express.json());
Then:
const endpointSecret = 'whsec_........'
const stripeWebhook = async (req, res) => {
const sig = req.headers['stripe-signature'];
let eventSecure = {}
try {
eventSecure = stripe.webhooks.constructEvent(req.body, sig, endpointSecret);
//console.log('eventSecure :', eventSecure);
}
catch (err) {
console.log('err.message :', err.message);
res.status(400).send(`Webhook Secure Error: ${err.message}`)
return
}
res.status(200).send({ received: true });
}
Here is code which is working for me:
app.use(bodyParser.json({
verify: function (req, res, buf) {
var url = req.originalUrl;
if (url.startsWith('/stripe')) {
req.rawBody = buf.toString();
}
}
}));
And then pass the req.rawBody for verification
stripe.checkWebHook(req.rawBody, signature);
Reference: https://github.com/stripe/stripe-node/issues/341
2 things to note:
pass req.rawBody instead of req.body to constructEvent
const event = stripe.webhooks.constructEvent(
req.rawBody,
sig,
STRIPE_WEBHOOK_SECRET
);
Make sure you're using the correct webhook secret. It's unique per webhook url!
2021 - Solution
I faced that error, and after a lot research I could not figure out the problem easily, but finally I could do it based in my architecture below:
//App.js
this.server.use((req, res, next) => {
if (req.originalUrl.startsWith('/webhook')) {
next();
} else {
express.json()(req, res, next);
}
});
//routes.js
routes.post(
'/webhook-payment-intent-update',
bodyParser.raw({ type: 'application/json' }),
//your stripe logic (Im using a controller, but wherever)
(req, res) => {
stripe.webhooks.constructEvent(...)
}
)
Two big warnings to pay attention:
Make sure to send the req.headers['stripe-signature']
Make sure that your endpointSecret is right, if not it will still saying the same error
Tips:
Test it locally by installing the Stripe CLI: https://stripe.com/docs/webhooks/test
Verify your key on stripe dashboard or you can also make sure if you have the right key by verifying you stripe log as below:
I hope it helps you. :)
// Use JSON parser for all non-webhook routes
app.use(
bodyParser.json({
verify: (req, res, buf) => {
const url = req.originalUrl;
if (url.startsWith('/api/stripe/webhook')) {
req.rawBody = buf.toString();
}
}
})
);
The above code will look fine for the above answers. But even I was made one mistake. After put the same thing I got the same error.
Finally, I've figured it out if you're configured body-parser below the rawBody code then it'll work.
Like this
// Use JSON parser for all non-webhook routes
app.use(
bodyParser.json({
verify: (req, res, buf) => {
const url = req.originalUrl;
if (url.startsWith('/api/stripe/webhook')) {
req.rawBody = buf.toString();
}
}
})
);
// Setup express response and body parser configurations
app.use(express.json());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
Hopefully, it'll help someone.
It is late but will help others
Github answer
const payload = req.body
const sig = req.headers['stripe-signature']
const payloadString = JSON.stringify(payload, null, 2);
const secret = 'webhook_secret';
const header = stripe.webhooks.generateTestHeaderString({
payload: payloadString,
secret,
});
let event;
try {
event = stripe.webhooks.constructEvent(payloadString, header, secret);
} catch (err) {
console.log(`Webhook Error: ${err.message}`)
return res.status(400).send(`Webhook Error: ${err.message}`);
}
switch (event.type) {
case 'checkout.session.completed': {
......
enter code here
If you are trying to add a stripe webhook into your NextJS API Route, here's how to do so (ref):
import initStripe from "stripe";
import { buffer } from "micro";
import { NextApiRequest, NextApiResponse } from "next";
export const config = { api: { bodyParser: false } };
const handler = async (req: NextApiRequest, res: NextApiResponse) => {
const stripe = initStripe(process.env.STRIPE_SECRET_KEY||'');
const signature = req.headers["stripe-signature"];
const signingSecret = process.env.STRIPE_WEBHOOK_SECRET || '';
const reqBuffer = await buffer(req);
let event;
try {
event = stripe.webhooks.constructEvent(reqBuffer, signature, signingSecret);
} catch (error: any) {
console.log(error);
return res.status(400).send(`Webhook error: ${error?.message}`);
}
console.log({ event });
res.send({ received: true });
};
export default handler;
This is using buffer from the micro library, in combination with the modifying the default API request to use request's rawbody. In some frameworks (like NextJs), rawBody doesn't come OOTB, hence the workaround of retrieving the rawbody by reqBuffer, which is needed in the stripe.webhooks.constructEvent event.
I was able to obtain data from one webhook but not from a second one: the problem was that the secret key I used was the same as the one used for the first webhook, but I found out that every webhook has a different key, that's way I got that same message.
AWS API Gateway + Lambda (Express.js CRUD) I'm using this for Stripe webhook endpoint and it works for me:
app.use(require('body-parser').text({ type: "*/*" }));
This happened to me when sending a test webhook from the Stripe dashboard after I had renamed a firebase cloud function. All my other functions were working fine. Solved by re-setting in the terminal
firebase functions:config:set stripe.webhook_signature="Your webhook signing secret"
(if you're using that) and redeploying the functions firebase deploy --only functions
On a second occasion I solved the problem by rolling the stripe signature in the stripe dashboard.
Please use this script
app.use(
bodyParser.json({
verify: (req, res, buf) => {
req.rawBody = buf;
},
})
);
My fave was combining two of above great answers.
Then you can use req.rawbody when you construct the event.
Replace "webhook" with whatever route you wish you have a raw body for.
app.use(
"/webhook",
express.json({
verify: (req, res, buf) => {
req.rawBody = buf.toString();
},
})
);
BEFORE
app.use(express.json());
Works well if you are using routes and controllers.
To use raw body in express with a specific endpoint in a seperated middleware, my solution is just enabling router to use express.raw for the webhook endpoint.
-node.js v12
-express.js v4.17.1
export const handleBodyRequestParsing = (router: Router): void => {
router.use('/your_webhook_endpoint', express.raw({ type: '*/*' }))
router.use(express.json({ limit: '100mb' }))
router.use(express.urlencoded({ extended: true }))
}
Here is the Quick Tip which may save your hours !
If you are adding express payment to your exciting express app sometimes you may already pass your request as json in the beginning of application by using express middleware app.use(json()); or any other middleware (Bodyparser for example).
If you are doing that then change that to omit your webhook url
Exmaple:
Assume your payment webhook url is /paments/webhhok
app.use((req, res, next) => {
if (req.originalUrl.includes("/payments/webhook")) {
next();
} else {
express.json()(req, res, next);
}
});
When using Stripe in Express, if you have the following line in your code;
app.use(express.json());
it is going to prevent you from providing the raw body to the Stripe even when you explicitly set "bodyParser.raw", which will throw an error. This was the reason my code failed. Finally sorted it out.
I tried all the solutions above and no one worked, and figured out that the only solution was not to use express at all for this endpoint. you just have to create another http function
export const webhook = functions.https.onRequest(async (req, res) => {
try {
const sig = req.headers['stripe-signature']
const endpointSecret = 'web_secret'
const event = stripe.webhooks.constructEvent(
req.rawBody,
sig,
endpointSecret
)
console.log(event.data.object)
res.status(200).send(event.data.object)
} catch (err) {
console.error('ocorreu um erro', err)
res.status(400).send(`Webhook Error: ${err.message}`)
}
})
I'm having issues connecting to mongodb when I am using a cloud watch event rule to trigger lambdas to keep them warm and also the same issue when I tried using serverless-plugin-warmup. Anyone have and ideas as to why this would be happening? Also I whitelist IP's for my database and use an Elastic IP for my lambda functions. Could the cloud watch event rules be causing the lambdas to use a different IP?
{"error":{"name":"MongoError","message":"no connection available"}}
I wrap my functions with the following to make sure the database is connected before running code
const mongoose = require('mongoose');
mongoose.Promise = global.Promise;
let cachedDB = null;
module.exports = fn => (...args) => {
const [, context] = args;
context.callbackWaitsForEmptyEventLoop = false;
if (cachedDB && cachedDB.readyState != 0 && cachedDB.readyState != 3) {
fn(...args);
} else {
mongoose.connect(process.env.MONGO_URI);
mongoose.connection.on('error', err => {
console.log('Connection Error');
console.log(err);
});
mongoose.connection.once('open', () => {
cachedDB = mongoose.connection;
fn(...args);
});
}
};
You need to handle warmup events as a special case in your handler. Basically, your handler should return right away when it is invoked via a warmup event.
In the example given in serverless-warmup-plugin, you can do it this way,
module.exports.lambdaToWarm = function(event, context, callback) {
// Immediate response for WarmUP plugin
if (event.source === 'serverless-plugin-warmup') {
console.log('WarmUP - Lambda is warm!')
return callback(null, 'Lambda is warm!')
}
// add lambda logic after
}
Notice that there is an if statement at the beginning to check if it's a warmup event. If it is, return successfully right away.
This check should be at the beginning so that it doesn't have to connect to MongoDB at all.
Is it possible to run alexa skill locally with ngrok instead AWS? I built a skill in AWS Lambda but I would rather use my own server. What can I do to run Alexa locally?
I tried https://github.com/alexa-js/alexa-app-server but it makes any sense because I would need to rewrite my whole code :( The better solution is http://docs.bespoken.tools/en/latest/tutorials/tutorial_lambda_nodejs/ but it isn't the best. It just works only for a wellcome intent and freezes after that :(
Terminal Logs from bespken command:
BST: v0.9.35 Node: v7.8.0
Your URL for Alexa Skill configuration:
https://proxy.bespoken.tools?node-id=33efccba-2246-477f-bbb8-2e1e510cce9d
INFO 2017-04-25T20:27:20.628Z Connected - proxy.bespoken.tools:5000
INFO 2017-04-25T20:27:26.812Z RequestReceived: POST /?node-id=33efccba-2246-477f-bbb8-2e1e510cce9d ID: 1493152039146
INFO 2017-04-25T20:27:26.815Z Forwarding localhost:10000
Current hour: 24
Warning: Application ID is not set
INFO 2017-04-25T20:27:27.939Z ResponseReceived ID: 1493152039146
INFO 2017-04-25T20:28:10.755Z RequestReceived: POST /?node-id=33efccba-2246-477f-bbb8-2e1e510cce9d ID: 1493152078963
INFO 2017-04-25T20:28:10.756Z Forwarding localhost:10000
Warning: Application ID is not set
INFO 2017-04-25T20:28:11.157Z ResponseReceived ID: 1493152078963
INFO 2017-04-25T20:28:51.073Z RequestReceived: POST /?node-id=33efccba-2246-477f-bbb8-2e1e510cce9d ID: 1493152113739
INFO 2017-04-25T20:28:51.073Z Forwarding localhost:10000
Warning: Application ID is not set
INFO 2017-04-25T20:28:51.995Z ResponseReceived ID: 1493152113739
Yes, there are several solutions for running your node lambda locally. I've been using node-lambda, for example. Like most solutions it is oriented to users who want to test locally and then easily deploy to AWS Lambda.
If you want to run them yourself, I would note that MS and IBM have made their implementations of lambda open-source (here's MS's and IBM's). I haven't actually tried it myself, and I would note that with AWS, GCP, and Azure all providing Lambda services for node the market for these is healthy and the lock-in is minimal so I feel less need to be able to run it myself than for something like Dynamo.
But I also recommend that you continue to pursue BST. I'm using some of my own pieces for testing my skills because I got started before I heard of their stuff, but what I have tried of their's (BSTAlexa) is very useful and I see that they provide some of the other pieces you need for easy and effective testing of your skill.
Here's some sample code that you can use to easily run a Lambda locally, call this file AlexaLambda.js:
const log = require('console');
var AWS = require('aws-sdk');
AWS.config.region = "us-east-1";
AWS.config.update({
accessKeyId: "----",
secretAccessKey: "----",
});
/**
* Wraps the actual underlying Alexa lambda initialization in a
* Promise. Injects test mocks where appropriate.
*/
var initializerPromise = new Promise(function(fulfill, reject) {
// Mock out certain imports here if you want but not necessary
/*
var Module = require('module');
var originalRequire = Module.prototype.require;
Module.prototype.require = function() {
if ((arguments[0] == 'S3FeedService') ||
(arguments[0] == './lib/S3FeedService')) {
return MockS3Service;
} else if ((arguments[0] == 'WebsocketService') ||
(arguments[0] == './lib/WebsocketService')) {
return WSMockService;
} else if ((arguments[0] == 'SQSService') ||
(arguments[0] == './lib/SQSService')) {
return SQSMockService;
} else {
return originalRequire.apply(this, arguments);
}
};*/
// Import your actual lambda here.
var lambda = require('../src/index.js');
fulfill(lambda);
});
/**
* The Alexa Lambda context object which is called upon completion
* of lambda execution. Also wraps the callback which contains the
* test assertion code of the caller.
* #param callback - must be of the form function(error, result) {};
* #returns
*/
function Context(callback) {
this.clientContext = {"env": {}};
this.callback = callback;
}
Context.prototype.done = function(error, result) {
if (typeof error != "undefined" && error) {
this.callback(error, null);
} else {
this.callback(null, result);
}
}
Context.prototype.succeed = function(result) {
this.callback(null, result);
}
Context.prototype.fail = function(error) {
this.callback(error, null);
}
/**
* The AlexaLambda object that's exposed for test cases.
* #returns
*/
function AlexaLambda() {
}
/**
* Executes the lambda function, provided an inputEvent and a
* callback.
* #param inputEvent - the input event that includes the intent.
* #param callback - called upon completion of lambda execution.
*/
AlexaLambda.prototype.execute = function(inputEvent, callback) {
initializerPromise.then(function(lambda) {
var context = new Context(callback);
lambda.handler(inputEvent, context);
});
}
/**
* Export the lambda class, importers instantiate via new AlexaLambda();
*/
module.exports = AlexaLambda;
Then you can use this 'AlexaLambda' in your tests like so (in my case, I'm using Mocha):
var AlexaLambda = require('./AlexaLambda');
var Event = require('./Event'); // My 'fake' Event class
describe("Guest User Test", function() {
var alexaLambda = new AlexaLambda();
it("Alexa, open/launch 60db", function(done) {
var event = Event.createLaunchEvent();
alexaLambda.execute(event, function(error, result) {
validateYourResultHere();
done();
})
});
Then it's just a matter of running your test via whatever framework you're using.
You can test your alexa skill locally by following the following tutorial:
How to test Alexa locally