I aware that if an PCIe endpoint send several read request to the host, the completion packets returned may not be in order, and then we need the tag field to reorder them.
But I want to know if there is another way to identify the CPLD TLP?
For example, if I send 2 read request of 128 Bytes with the same tag to the host, by return, if the CPLDs don't arrive in order, then is there any chance that I can identify and reorder them?
Abbreviations:
TLP - Transaction Layer Protocol
CPLD - Completion Data
CPLDs can not get out of order, as long as they are send with the same traffic class (TC), which feeds all packets through the same virtual channel (VC). Reordering is done between different VCs with different priority. So 2 read requests will be answered in the same order. A responder might combine both reads to one completion if you are addressing consecutive addresses!
As per PCI EXPRESS BASE SPECIFICATION , PCIe CPLD Ordering for same Transaction ID must be in order Refer Transaction Ordering Table 2-34, rule D5b, CPLD must be in order.
Related
I have a list of transactions which I periodically fetch via Solana API. Out of those, I need to identify the ones that are:
incoming that transfer SOL
incoming that transfer a token (USDT, USDC, may be 1-2 more)
That is, I need to implement a simple way of watching "top up" TX-s, for SOL and the stablecoins, for a list of accounts of mine.
So how to clearly identify the type of transfer of a TX I'm parsing: whether it's EITHER SOL transfer, OR a token transfer? As well as the amount.
Will it be by the presense of the field lamports - SOL transfer. Namely, instructions[X].parsed.info["lamports"] Will be be a reliable way?
On the contrary: how to identify if a TX is a pure "token transfer" one? Not the one that may involve "token transfer" among other instructions, but is only "token transfer" one.
I don't expect any other, more complex types of TX in my list of addresses. If such TX-s happen to occur, I'll filter them out during parsing.
Assume I started a transaction yesterday which is not yet confirmed (In Pending status).
Later four other transactions are Successful and assumed they are having block numbers
1110, 1111, 1112, 1113.
Assume the old transaction got confirmed at this point in time. May I know what can be the block number of that old one? Can it be less than 1110 or it will be greater than 1113.
I tested but my transactions are fast and unable to generate this scenario.
The reason for asking is, to read ether scan data using block numbers.
The tx will be in a higher block (Assuming it gets accepted by a miner at some point). It is currently in the mempool waiting to be mined. You can query the blockchain to get the status (Depends on client API, special clients like Alchemy and QuickNode may have special tools to explore the mempool e.g. Alchemy have a mempool watcher).
This is a good primer to understand what is happening.
Apparently (based on an exception) EventHubClient.SendBatch and EventHubClient.SendBatchAsync only support sending to a single partition per operation. This appears to be indicated indicated in the documentation by the method summary "Sends a batch of event data to the logical partition represented by PartitionId" which appears to be copied from the partition specific EventHubSender.SendBatch.
Are there design considerations (vs just writing less code) in having the higher level client not rebatch as needed?
The EventHubClient has control over the partition key hashing/distribution which is not available to callers of EventHubClient that wish to send a batch of data with differing keys that may lie on the same partition. Left to rebatch myself I need to make calls on the order of the number of messages as opposed to on the number of partitions which with small messages is easily two orders of magnitude difference.
Since it's already necessary to rebatch it could be worse.
I was assuming the PartitionKey on your EventData objects of the batch would be used to partition out. But apparently not.
However, there's Paolo Salvatori, who wrote a nice set of Extension methods to provide good and easy support for sending in batches to Event Hub.
You'll probably like his post here: http://blogs.msdn.com/b/paolos/archive/2015/03/26/how-to-implement-a-partitioned-sendbatch-method-for-azure-service-bus-entities.aspx
Best regards
In an attempt to use Dynamodb for one of projects, I have a doubt regarding the strong consistency model of dynamodb. From the FAQs
Strongly Consistent Reads — in addition to eventual consistency, Amazon DynamoDB also gives you the
flexibility and control to request a strongly consistent read if your application, or an element of your application, requires it. A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.
From the definition above, what I get is that a strong consistent read will return the latest write value.
Taking an example: Lets say Client1 issues a write command on Key K1 to update the value from V0 to V1. After few milliseconds Client2 issues a read command for Key K1, then in case of strong consistency V1 will be returned always, however in case of eventual consistency V1 or V0 may be returned. Is my understanding correct?
If it is, What if the write operation returned success but the data is not updated to all replicas and we issue a strongly consistent read, how it will ensure to return the latest write value in this case?
The following link
AWS DynamoDB read after write consistency - how does it work theoretically? tries to explain the architecture behind this, but don't know if this is how it actually works? The next question that comes to my mind after going through this link is: Is DynamoDb based on Single Master, multiple slave architecture, where writes and strong consistent reads are through master replica and normal reads are through others.
Short answer: Writing successfully in strongly consistent mode requires that your write succeed on a majority of servers that can contain the record, therefore any future consistent reads will always see the same data, because a consistent read must read a majority of the servers that can contain the desired record. If you do not perform a strongly consistent read, the system will ask a random server for the record, and it is possible that the data will not be up-to-date.
Imagine three servers. Server 1, server 2 and server 3. To write a strongly consistent record, you pick two servers at minimum, and write the data. Let's pick 1 and 2.
Now you want to read the data consistently. Pick a majority of servers. Let's say we picked 2 and 3.
Server 2 has the new data, and this is what the system returns.
Eventually consistent reads could come from server 1, 2, or 3. This means if server 3 is chosen by random, your new write will not appear yet, until replication occurs.
If a single server fails, your data is still safe, but if two out of three servers fail your new write may be lost until the offline servers are restored.
More explanation:
DynamoDB (assuming it is similar to the database described in the Dynamo paper that Amazon released) uses a ring topology, where data is spread to many servers. Strong consistency is guaranteed because you directly query all relevant servers and get the current data from them. There is no master in the ring, there are no slaves in the ring. A given record will map to a number of identical hosts in the ring, and all of those servers will contain that record. There is no slave that could lag behind, and there is no master that can fail.
Feel free to read any of the many papers on the topic. A similar database called Apache Cassandra is available which also uses ring replication.
http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf
Disclaimer: the following cannot be verified based on the public DynamoDB documentation, but they are probably very close to the truth
Starting from the theory, DynamoDB makes use of quorums, where V is the total number of replica nodes, Vr is the number of replica nodes a read operation asks and Vw is the number of replica nodes where each write is performed. The read quorum (Vr) can be leveraged to make sure the client is getting the latest value, while the write quorum (Vw) can be leveraged to make sure that writes do not create conflicts.
Based on the fact that there are no write conflicts in DynamoDB (since these would have to be reconciliated from the client, thus being exposed in the API), we conclude that DynamoDB is using a Vw that respects the second law (Vw > V/2), probably just V/2+1 to reduce write latency.
Now regarding read quorums, DynamoDB provides 2 different kinds of read. The strongly consistent read uses a read quorum that respects the first law (Vr + Vw > V), probably just V/2 if we assume V/2+1 for writes as before. However, an eventually consistent read can use only a single random replica Vr = 1, thus being much quicker but giving zero guarantee around consistency.
Note: There's a possibility that the write quorum used does not respect the second law (Vw > V/2), but that would mean DynamoDB resolves automatically such conflicts (e.g. by selecting the latest one based on local time) without reconciliation from the client. But, I believe that this is really unlikely to be true, since there is no such reference in the DynamoDB documentation. Even in that case though, the rest reasoning stays the same.
You can find answer to your question here: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/APISummary.html
When you issue a strongly consistent read request, Amazon DynamoDB returns a response with the most up-to-date data that reflects updates by all prior related write operations to which Amazon DynamoDB returned a successful response.
In your example, if the updateItem request to update the value from v0 to v1 was successful, the subsequent strongly consistent read request will return v1.
Hope this helps.
I have the below situation which need to be addressed efficiently,
I'm doing file sync from client devices to server. Sometimes what happen is file from one device doesn't get fetched to another device from the server due to some issues with server. I need to make sure that all the files in the server are synced to all the client devices using a separate thread. I am using C++ for the development and libcurl for client to server communication.
Here in the client device, we have an entry for downloaded files in the SQLite Database. Likewise in the server, we have similar updates in the server databases (MySQL) too. I need to list all the available files from the client device and send it to server and have to compare it with the list taken from the server database to find out the missed files.
I did a rough estimation that for 1 million files list (File Name with Full Path), it is about 85 MB in size. Upon compression it goes upto 10 MB in size. So transferring this entire file list (even after compression) from client to server is not a good idea. I planned to implement Bloom Filters for this as below,
Fetch files list from client side database and convert those to Bloom Filter Data Structure.
Just transferring the bloom data structure alone from client to the server.
Fetch files list from server side database and compare it with Bloom data structure received from the client and find out the missing files.
Please note that the above process initiated from client should be handled in thread at regular interval say for every 1 hour or so.
The problem with Bloom filters is false positive rates even if it very low. I don't want to miss out even a single file. Is there any other better way of doing this ?.
As you've noticed, this isn't a problem for which Bloom Filters are appropriate. With a Bloom Filter, when you get a hit you must then check the authoritative source to differentiate between a false positive and a true positive - they're useful in situations where most queries against the filter will be expected to give a negative result, which is the opposite to your case.
What you could do is have each side build a partial Prefix Tree in memory of all the filenames known to that side. It wouldn't be a full prefix tree - once you number of filenames below a node drops below a certain level you'd just include the full list of those filenames in that node. You then synchronise those prefix trees using a recursive algorithm starting at the root of the trees:
Each side creates a hash of all the sorted, concatenated filenames below the current node.
If the hashes are equal then this node and all descendents are synchronised - return.
If there are no child nodes, send the (short) list of filenames at this terminal node from one side to the other to synchronise and return.
Otherwise, recursively synchronise the child nodes and return.
The hash should be at least 128 bits, and make sure that when you concatenate the filenames for the hash you do so in a reversible manner (ie. seperate them with a character that can't appear in filenames like \0, or prefix each one with its length).
In file/pathname compression I've found a prefix-suffix compression to work better even alone than a generic (bz2) compression. When combined, the filename list could be reduced even more.
The trick is in using escape codes (e.g. <32) to indicate the number of common characters to the previous row, then use regular characters for the unique part and finally (optionally) encode the number of common characters at the end of the string.