I'm doing an ad server (sort of RTB), there are advertisers who pay to promote their ad campaigns.
When the user watched the ad I want to charge the advertiser.
Ad campaigns should participate in auctions without blocking, means he can bid for multiple ad requests at the same time. It would make it difficult to charge advertiser immediately since I'll have to block his balance.
Another approach is to not charge him immediately, but in the separate process once every N seconds, and hope so he didn't buy more impressions than he can afford. I can make some sort of threshold credit he should have to participate in auctions, it would eliminate most of the overruns but what if the process is exited and the advertiser wasn't charged and overrun a lot, that would be a problem.
Can someone please advise me how these things are usually handled, maybe recommend some book/article on the topic, please?
If this is a problem of asynchronous programming, meaning that you are concerned about overcharging a bidder when they place too many impression bids at once. Then I suggest using a mutex (i.e. locking system). You can set a threshold on the number of locks a bidder can have in your system at any given time. This threshold can be equivalent to the maximum amount the bidder is willing to spend on impressions. When the bidder requests a bid a lock for that bid on his account should be sent to your server. When the server responds with the ok response that the lock has been created, then the bid can take place. The bid is active until the lock is released by your server. This makes it so the server keeps track of all the bids and places locks on all of them. If the bidder has a threshold of 10 bids and he tries to make a bid for the 11th the server will not release a lock to the bidder to make this bid. Furthermore, if you are using a microservice architecture I suggest making a transaction manager service to handle all these requests and even use Saga based transactions for rollback functionally in case of a server failure.
Related
Is it possible to pause a initiated flashloan and resume after some parameter met.
I read on solidity docs, all function are require some sort of user action to execute.
Just incase I miss out if there any method to achieve it.
A flashloan needs to be repaid in the same transaction in which it was initiated. If it's not repaid, the whole transaction reverts as if it never happened (however the transaction sender still pays the gas fees).
So it is not really possible to pause and resume a flash loan, as it either happens all in one instance - or it doesn't.
The user action is usually executing one contract, that performs several internal calls within the same transaction:
initiate the flash loan
perform some kind of trade with the loaned funds
give aproval to the flash loan contract to pull the funds back
repay the flash loan
I have a client which has a pretty popular ticket selling service, to the point that the microservice based backend is struggling to keep up, I need to come up with a solution to optimize and loadbalance the system. The infrastructure works through a series of interconnected microservices.
When a user enter the sales channels (mobile or web app), the request is directed to an AWS API Gateway which is in charge of orchestrating the communication towards the microservice in charge of obtaining the requested resources.
These resources are provided from a third party API
This third party has physical servers in each venue in charge of synchronizing the information between the POS systems and the digital sales channels.
We have a REDIS instance in charge of caching these requests that we make to the third party API, we cache each endpoint with a TTL relative to the frequency of updating the information.
Here is some background info:
We get traffic mostly from 2 major countries
On a normal day, about 100 thousands users will use the service, with an 70%/30% traffic relation in between the two countries
On important days, each country has different opening hours (Country A starts sales at 10 am UTC, but country B starts at 5 pm UTC), on these days the traffic increases some n times
We have a main MiddleWare through which all requests made by clients are processed.
We have a REDIS cache database that stores GETs with different TTLs for each endpoint.
We have a MiddleWare that decides to make the request to the cache or to the third party's API, as the case may be.
And these are the complaints I have gotten that need to be deal with:
When a country receives a high amount of requests, the country with the least traffic gets negatively affected, the clients do not respond, or respond partially because the computation layer's limit was exceeded and so the users have a bad experience
Every time the above happens, the computation layer must be manually increased from the infrastructure.
Each request has different response times, stadiums respond in +/- 40 seconds and movie theaters in 3 seconds. These requests enter a queue and are answered in order of arrival.
The error handling is not clear. The errors are mixed up and you can't tell from which country the errors are coming from and how many errors there are
The responses from the third party API are not cached correctly in the cache layer since errors are stored for the time of the TTL
I was thinking of a couple of thinks that I could suggest:
Adding in instrumentation of the requests by using AWS X-Ray
Adding in a separate table for errors in the redis cache layer (old data has to be better than no data for the end user)
Adding in AWS elastic load balancing for the main middleware
But I'm not sure how realistic would be to implement these 3 things, I'm also not sure if they would even solve the problem, I personally don't really have experience with optimizing this type of backed. I would appreciate any suggestions, recommendations, links, documentation, etc. I'm really desperate for a solution to this problem
few thoughts:
When a country receives a high amount of requests, the country with the least traffic gets negatively affected, the clients do not respond, or respond partially because the computation layer's limit was exceeded and so the users have a bad experience
A common approach in aws is to regionalize stack - assuming you are using cdk/cloud formation creating regionalized stack should be a straightforward task.
But it is a question if this will solve the problem. Your system suffers from availability issues, regionalization will isolate this problem down to regions. So we should be able to do better (see below)
Every time the above happens, the computation layer must be manually increased from the infrastructure.
AWS has an option to automatically scale up and down based on traffic patterns. This is a neat feature, given you set limits to make sure you are not overcharged.
Each request has different response times, stadiums respond in +/- 40 seconds and movie theaters in 3 seconds. These requests enter a queue and are answered in order of arrival.
It seems that the large variance is because you have to contact the servers at venues. I recommend to decouple that activity. Basically calls to venues should be done async; there are several ways you could do that - queues and customer push/pull are the approaches (please, comment if more details are needed. but this is quite standard problem - lots of data in the internet)
The error handling is not clear. The errors are mixed up and you can't tell from which country the errors are coming from and how many errors there are
That's a code fix, when you do send data to cloudwatch (do you?). You could put country as a context to all request, via filter or something. And when error is logged that context is logged as well. You probably need venue id even more than country, as you can conclude country from venue id.
The responses from the third party API are not cached correctly in the cache layer since errors are stored for the time of the TTL
Don't store errors + add a circuit breaker pattern.
In the user scenario, for a running auction there can be many user bids which need to be handled in the order they arrive. In my tests natural option is to go with ServiceBus Queues with support FIFO properly.
Few Issues with this as I see is;
When there are lot of parallel auctions (Auction for itemA, B, C etc..), It's do not think it's feasible to create different queues per auction. But then pushing bids to a single queue will also have a bottle neck.
Topics does not guarantee FIFO. But according to this using SupportOrdering can work (yet to test).
I was wondering whether eventhub can be used to model this scenario? At the end of the day there should be a layer that publishes bids ASAP and there are couple of workers taking action on those bids in order (and forward them to other sub systems)
Has anyone out there tried to tackle similar use case? Our current bid performance baseline is like (less than 200 bids per second for the moment)
Event Hubs guarantees ordering of events when reading from a partition, though not reading events from different partitions. Assuming that you were sending events for a given auction to a single partition, your ordering needs would be satisfied.
The one additional thing that I'd like to mention is expectations of fairness in this kind of competitive scenario. An Event Hubs partition will preserve ordering with respect to the order in which they were received by the broker. That may not agree with the perceived order that users submit the bid due to network latency or transient failures that require retries.
I am new to blockchain and have a question about its transaction verification process. I have read some articles related to Bitcoin. They said it takes about 10 minutes to verify a bitcoin transaction. Does this mean if I purchase a bitcoin I have to wait 10 minutes for this transaction? If yes, how can I handle the real-time system requirement in the blockchain?
They said it takes about 10 minutes to verify a bitcoin transaction.
Does this mean if I purchase a bitcoin I have to wait 10 minutes for
this transaction?
Yes if your transaction was lucky enough (or expensive enough) to make it into the next block. Since Bitcoin's 10-minute blocks hold a finite number of transactions, you might need to wait longer than that if your transaction wasn't included in the upcoming block.
On top of that, once your transaction is on the blockchain, you should ideally wait until several more blocks are mined until you're confident that your transaction got included in the globally-accepted branch (the longest branch).
So, to be safe, if you decide to wait for 6 blocks after your transaction is included, you'd end up waiting an additional hour until you're relatively confident that your transaction has been accepted.
If yes, how can I handle the real-time system requirement in the
blockchain?
That's one challenge that Bitcoin hasn't overcome yet. The closest work-in-progress solution is the Lightning Network, but that's still in development.
In the meantime, some applications might decide to process real-time transactions off-chain by accepting transactions before they're included in the blockchain. But this is a risky trade-off; you'd allow fast transactions at the cost of security, opening yourself up to double-spending.
Imagine 3 system components:
1. External ecommerce web service to process credit card transactions
2. Local Database to store processing results
3. Local UI (or win service) to perform payment processing of the customer order document
The external web service is obviously not transactional, so how to guarantee:
1. results to be eventually persisted to database when received from web service even in case the database is not accessible at that moment(network issue, db timeout)
2. prevent clients from processing the customer order while payment initiated by other client but results not successfully persisted to database yet(and waiting in some kind of recovery queue)
The aim is to do processing having non transactional system components and guarantee the transaction won't be repeated by other process in case of failure.
(please look at it in the context of post sell payment processing, where multiple operators might attempt manual payment processing; not web checkout application)
Ask the payment processor whether they can detect duplicate transactions based on an order ID you supply. Then if you are unable to store the response due to a database failure, you can safely resubmit the request without fear of double-charging (at least one PSP I've used returned the same response/auth code in this scenario, along with a flag to say that this was a duplicate).
Alternatively, just set a flag on your order immediately before attempting payment, and don't attempt payment if the flag was already set. If an error then occurs during payment, you can investigate and fix the data at your leisure.
I'd be reluctant to go down the route of trying to automatically cancel the order and resubmitting, as this just gets confusing (e.g. what if cancelling fails - should you retry or not?). Best to keep the logic simple so when something goes wrong you know exactly where you stand.
In any system like this, you need robust error handling and error reporting. This is doubly true when it comes to dealing with payments, where you absolutely do not want to accidentaly take someone's money and not deliver the goods.
Because you're outsourcing your payment handling to a 3rd party, you're ultimately very reliant on the gateway having robust error handling and reporting systems.
In general then, you hand off control to the payment gateway and start a task that waits for a response from the gateway, which is either 'payment accepted' or 'payment declined'. When you get that response you move onto the next step in your process and everything is good.
When you don't get a response at all (time out), or the response is invalid, then how you proceed very much depends on the payment gateway:
If the gateway supports it send a 'cancel payment' style request. If the payment cancels successfully then you probably want to send the user to a 'sorry, please try again' style page.
If the gateway doesn't support canceling, or you have no communications to the gateway then you will need to manually (in person, such as telephone) contact the 3rd party to discover what went wrong and how to proceed. To aid this you need to dump as much detail as you have to error logs, such as date/time, customer id, transaction value, product ids etc.
Once you're back on your site (and payment is accepted) then you're much more in control of errors, but in brief if you cant complete the order, then you should either dump the details to disk (such as csv file for manual handling) or contact the gateway to cancel the payment.
Its also worth having a system in place to track errors as they occur, and if an excessive number occur then consider what should happen. If its a high traffic site for example you may want to temporarily prevent further customers from placing orders whilst the issue is investigated.
Distributed messaging.
When your payment gateway returns submit a message to a durable queue that guarantees a handler will eventually get it and process it. The handler would update the database. Should failure occur at that point the handler can leave the message in the queue or repost it to the queue, or post an alternate message.
Should something occur later that invalidates the transaction, another message could be queued to "undo" the change.
There's a fair amount of buzz lately about eventual consistency and distribute messaging. NServiceBus is the new component hotness. I suggest looking into this, I know we are.