Hi i am making a finance application that depends on Events from blockchain,
Basically i update my database based on Events i receive using web3js, and when user asks i sign with private key for the contract to be able to give user Money.
My only concern is can i depend on events? like can there be a case where i miss events?
here is my code for doing so :
const contract = new web3.eth.Contract(abi, contract_address)
const stale_keccak256 = "0x507ac39eb33610191cd8fd54286e91c5cc464c262861643be3978f5a9f18ab02";
const unStake_keccak256 = "0x4ac743692c9ced0a3f0052fb9917c0856b6b12671016afe41b649643a89b1ad5";
const getReward_keccak256 = "0x25c30c62c42b51e4f667b70ef60f1f683c376f6ace28312ed45a40665e01af37";
let userRepository: Repository<UserData> = connection.getRepository(UserData);
let globalRepository: Repository<GlobalStakingInfo> = connection.getRepository(GlobalStakingInfo);
let userStakingRepository: Repository<UserStakingInfo> = connection.getRepository(UserStakingInfo);
let transactionRepository: Repository<Transaction> = connection.getRepository(Transaction);
const topics = []
web3.eth.subscribe('logs', {
address: contract_address, topics: topics
},
function (error: Error, result: Log) {
if (error) console.log(error)
}).on("data", async function (log: Log) {
let response: Boolean = false;
try {
response = await SaveTransaction(rpc_url, log.address, log.transactionHash, transactionRepository)
} catch (e) {
}
if (response) {
try {
let global_instance: GlobalStakingInfo | null = await globalRepository.findOne({where: {id: 1}})
if (!global_instance) {
global_instance = new GlobalStakingInfo()
global_instance.id = 1;
global_instance = await globalRepository.save(global_instance);
}
if (log.topics[0] === stale_keccak256) {
await onStake(web3, log, contract, userRepository, globalRepository, userStakingRepository, global_instance);
} else if (log.topics[0] === unStake_keccak256) {
await onUnStake(web3, log, contract, userStakingRepository, userRepository, globalRepository, global_instance)
} else if (log.topics[0] === getReward_keccak256) {
await onGetReward(web3, log, userRepository)
}
} catch (e) {
console.log("I MADE A BOBO", e)
}
}
}
)
The Code works and everything, i am just concerned if i could maybe miss a event? cause finance is related and people will lose money if missing event is a thing.
Please advise
You can increase redundancy by adding more instances of the listener connected to other nodes.
And also by polling past logs - again recommended to use a separate node.
Having multiple instances doing practically the same thing will result in multiplication of incoming data, so don't forget to store only unique logs. This might be a bit tricky, because theoretically one transaction ID can produce the same log twice (e.g. through a multicall), so the unique key could be a combination of a transaction ID as well as the log index (unique per block).
Related
This is my first app with Dart/Flutter/Riverpod, so any advice or comment about the code is welcome.
I'm using Hive as embedded db so the initial value for the provider's state is loaded asynchronously and using an AsyncValue of riverpod to wrapped it.
The following code works but I've got some doubts about the testing approach, so I would like to confirm if I'm using the Riverpod lib as It supposed to be used.
This is my provider with its deps (Preferences is a HiveObject to store app general config data):
final hiveProvider = FutureProvider<HiveInterface>((ref) async {
return await App.setUp();
});
final prefBoxProvider = FutureProvider<Box<Preferences>>((ref) async {
final HiveInterface hive = await ref.read(hiveProvider.future);
return hive.openBox<Preferences>("preferences");
});
class PreferencesNotifier extends StateNotifier<AsyncValue<Preferences>> {
late Box<Preferences> prefBox;
PreferencesNotifier(Future<Box<Preferences>> prefBoxFuture): super(const AsyncValue.loading()) {
prefBoxFuture.then((value) {
prefBox = value;
_loadCurrentPreferences();
});
}
void _loadCurrentPreferences() {
Preferences pref = prefBox.get(0) ?? Preferences();
state = AsyncValue.data(pref);
}
Future<void> save(Preferences prefs) async {
await prefBox.put(0, prefs);
state = AsyncValue.data(prefs);
}
Preferences? get preferences {
return state.when(data: (value) => value,
error: (_, __) => null,
loading: () => null);
}
}
final preferencesProvider = StateNotifierProvider<PreferencesNotifier, AsyncValue<Preferences>>((ref) {
return PreferencesNotifier(ref.read(prefBoxProvider.future));
});
And the following is the test case, I'm mocking the Hive box provider (prefBoxProvider):
class Listener extends Mock {
void call(dynamic previous, dynamic value);
}
Future<Box<Preferences>> prefBoxTesting() async {
final hive = await App.setUp();
Box<Preferences> box = await hive.openBox<Preferences>("testing_preferences");
await box.clear();
return box;
}
void main() {
test('Preferences value changes', () async {
final container = ProviderContainer(overrides: [
prefBoxProvider.overrideWithValue(AsyncValue.data(await prefBoxTesting()))
],);
addTearDown(() {
container.dispose();
Hive.deleteBoxFromDisk("testing_preferences");
});
final listener = Listener();
container.listen<AsyncValue<Preferences>>(
preferencesProvider,
listener,
fireImmediately: true,
);
verify(listener(null, const TypeMatcher<AsyncLoading>())).called(1);
verifyNoMoreInteractions(listener);
// Next line waits until we have a value for preferences attribute
await container.read(preferencesProvider.notifier).stream.first;
verify(listener(const TypeMatcher<AsyncLoading>(), const TypeMatcher<AsyncData>())).called(1);
Preferences preferences = Preferences.from(container.read(preferencesProvider.notifier).preferences!);
preferences.currentListName = 'Lista1';
await container.read(preferencesProvider.notifier).save(preferences);
verify(listener(const TypeMatcher<AsyncData>(), const TypeMatcher<AsyncData>())).called(1);
verifyNoMoreInteractions(listener);
final name = container.read(preferencesProvider.notifier).preferences!.currentListName;
expect(name, equals('Lista1'));
});
}
I've used as reference the official docs about testing Riverpod and the GitHub issue related with AsyncValues
Well, I found some problems to verify that the listener is called with the proper values, I used the TypeMatcher just to verify that the state instance has got the proper type and I check ("manually") the value of the wrapped object's attribute if It's the expected one. Is there a better way to achieve this ?
Finally, I didn't find too many examples with StateNotifier and AsyncValue as state type, Is there a better approach to implement providers that are initialized with deferred data ?
I didn't like too much my original approach so I created my own Matcher to compare wrapped values in AsyncValue instances:
class IsWrappedValueEquals extends Matcher {
final dynamic value;
IsWrappedValueEquals(this.value);
#override
bool matches(covariant AsyncValue actual, Map<dynamic, dynamic> matchState) =>
equals(actual.value).matches(value, matchState);
#override
Description describe(Description description) => description.add('Is wrapped value equals');
}
In the test, the final part is a bit different:
Preferences preferences = Preferences.from(container.read(preferencesProvider.notifier).preferences!);
preferences.currentListName = 'Lista1';
await container.read(preferencesProvider.notifier).save(preferences);
// the following line is the new one
verify(listener(IsWrappedValueEquals(Preferences()), IsWrappedValueEquals(preferences))).called(1);
verifyNoMoreInteractions(listener);
}
I prefer my custom Matcher to the original code, but I feel that there are too many custom code to test something, apparently, common.
If anyone can tell me a better solution for this case, It'd be great.
We use unmanaged transactions in many cases but this issue only occurs in 2 functions which are invoked more commonly and only on production environment (Haven't been able to reproduce it on dev).
Our code looks similar to this:
const t = await database.t();
try {
await function1(..., {t});
await function2(..., {t});
}
catch (e) {
await t.rollback(); <- This throws the error
throw e
};
// more logic
await t.commit();
Where database is just a Sequelize instantiation with name of database, username and password.
We assumed it's a connection error according to this: https://github.com/sequelize/sequelize/issues/4850 but in this case code wouldn't reach beyond line one on our pseudocode.
Error output on CloudWatch:
If you have multiple try-catch then maybe need to check the transaction state before rollback/commit:
const t = await database.t();
try {
await function1(..., { t });
await function2(..., { t });
}
catch (e) {
if (!t.finished) {
await t.rollback();
}
throw e;
};
// more logic
await t.commit();
Reference links:
https://github.com/sequelize/sequelize/issues/6547#issuecomment-466016971
https://github.com/sequelize/sequelize/pull/5043/files#diff-6c5ddfc7d68c447e32ef4c38b7ed69628910355ea0aff735bd4bcecc1256a8d8
We're trying to develop a self-invoking lambda to process S3 files in chunks. The lambda role has the policies needed for the invocation attached.
Here's the code for the self-invoking lambda:
export const processFileHandler: Handler = async (
event: S3CreateEvent,
context: Context,
callback: Callback,
) => {
let bucket = loGet(event, 'Records[0].s3.bucket.name');
let key = loGet(event, 'Records[0].s3.object.key');
let totalFileSize = loGet(event, 'Records[0].s3.object.size');
const lastPosition = loGet(event, 'position', 0);
const nextRange = getNextSizeRange(lastPosition, totalFileSize);
context.callbackWaitsForEmptyEventLoop = false;
let data = await loadDataFromS3ByRange(bucket, key, nextRange);
await database.connect();
log.debug(`Successfully connected to the database`);
const docs = await getParsedDocs(data, lastPosition);
log.debug(`upserting ${docs.length} records to database`);
if (docs.length) {
try {
// upserting logic
log.debug(`total documents added: ${await docs.length}`);
} catch (err) {
await recurse(nextRange.end, event, context);
log.debug(`error inserting docs: ${JSON.stringify(err)}`);
}
}
if (nextRange.end < totalFileSize) {
log.debug(`Last ${context.getRemainingTimeInMillis()} milliseconds left`);
if (context.getRemainingTimeInMillis() < 10 * 10 * 10 * 6) {
log.debug(`Less than 6000 milliseconds left`);
log.debug(`Invoking next iteration`);
await recurse(nextRange.end, event, context);
callback(null, {
message: `Lambda timed out processing file, please continue from LAST_POSITION: ${nextRange.start}`,
});
}
} else {
callback(null, { message: `Successfully completed the chunk processing task` });
}
};
Where recurse is an invocation call to the same lambda. Rest of the things work as expected it just times out whenever the call stack comes on this invocation request:
const recurse = async (position: number, event: S3CreateEvent, context: Context) => {
let newEvent = Object.assign(event, { position });
let request = {
FunctionName: context.invokedFunctionArn,
InvocationType: 'Event',
Payload: JSON.stringify(newEvent),
};
let resp = await lambda.invoke(request).promise();
console.log('Invocation complete', resp);
return resp;
};
This is the stack trace logged to CloudWatch:
{
"errorMessage": "connect ETIMEDOUT 63.32.72.196:443",
"errorType": "NetworkingError",
"stackTrace": [
"Object._errnoException (util.js:1022:11)",
"_exceptionWithHostPort (util.js:1044:20)",
"TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)"
]
}
Not a good idea to create a self-invoking lambda function. In case of an error (could also be a bad handler call on AWS side) a lambda function might re-run several times. Very hard to monitor and debug.
I would suggest using Step Functions. I believe this tutorial can help Iterating a Loop Using Lambda
From the top of my head, if you prefer not dealing with Step Functions, you could create a Lambda trigger for an SQS queue. Then you pass a message to the queue if you want to run the lambda function another time.
I'm using apollo link in schema stitching as an access control layer. I'm not quite sure how to make the link return error response if a user does not have permissions to access a particular operation. I know about such packages as graphql-shield and graphql-middleware but I'm curious whether it's possible to achieve basic access control using apollo link.
Here's what my link looks like:
const link = setContext((request, previousContext) => merge({
headers: {
...headers,
context: `${JSON.stringify(previousContext.graphqlContext ? _.omit(previousContext.graphqlContext, ['logger', 'models']) : {})}`,
},
})).concat(middlewareLink).concat(new HttpLink({ uri, fetch }));
The middlewareLink has checkPermissions that returns true of false depending on user's role
const middlewareLink = new ApolloLink((operation, forward) => {
const { operationName } = operation;
if (operationName !== 'IntrospectionQuery') {
const { variables } = operation;
const context = operation.getContext().graphqlContext;
const hasAccess = checkPermissions({ operationName, context, variables });
if (!hasAccess) {
// ...
}
}
return forward(operation);
});
What should I do if hasAccess is false. I guess I don't need to forward the operation as at this point it's clear that a user does not have access to it
UPDATE
I guess what I need to do is to extend the ApolloLink class, but so far I didn't manage to return error
Don't know if anyone else needs this, but I was trying to get a NetworkError specifically in the onError callback using Typescript and React. Finally got this working:
const testLink = new ApolloLink((operation, forward) => {
let fetchResult: FetchResult = {
errors: [] // put GraphQL errors here
}
let linkResult = Observable.of(fetchResult).map(_ => {
throw new Error('This is a network error in ApolloClient'); // throw Network errors here
});
return linkResult;
});
Return GraphQL errors in the observable FetchResult response, while throwing an error in the observable callback will produce a NetworkError
After some digging I've actually figured it out. But I'm not quite sure if my approach is correct.
Basically, I've called forward with a subsequent map where I return an object containing errors and data fields. Again, I guess there's a better way of doing this (maybe by extending the ApolloLink class)
const middlewareLink = new ApolloLink((operation, forward) => {
const { operationName } = operation;
if (operationName !== 'IntrospectionQuery') {
const { variables } = operation;
const context = operation.getContext().graphqlContext;
try {
checkPermissions({ operationName, context, variables });
} catch (err) {
return forward(operation).map(() => {
const error = new ForbiddenError('Access denied');
return { errors: [error], data: null };
});
}
}
return forward(operation);
});
When someone makes a transfer from my account, an event gets trigger name
event Transfer(address indexed from, address indexed to, uint to)
Now I want to get notified when this event occurs on smart contract. I tried with difference things like filter, watch, subscription and etc. But nothing works as per need.
I also have an another query
What does filter, subscribe, and watch exactly do. I am always getting confuse between these terms. Can someone give a clear idea.
Note: I am using WEB3JS 1.0.0.26 version.
Here's a simple example for web3js 1.0.0.beta*:
function handler (event) {
console.log(event.returnValues);
}
function errorCallback (err) {
console.error(err);
}
let subscription = contractObj.events.TestEvent().subscription;
subscription.on('data', handler).on('error', errorCallback);
To unsubscribe:
subscription.unsubscribe(function (result) {
console.log(result)
});
Example of usage in class:
class Listener {
constructor(event, handler) {
this.subscription = event;
this.subscription.on('data', handler).on('error', this.errorCallback);
}
errorCallback(err) {
console.log(err);
}
}
class Test {
constructor(contractObj) {
this.contractObj = contractObj;
this.createListener();
}
createListener() {
let self = this;
this.listener = new Listener(this.contractObj.events.TestEvent(), function (event) {
self.returnValues = event.returnValues;
});
}
}