For a newly allocated mbuf, the area at which the data begins in the message buffer is RTE_PKTMBUF_HEADROOM bytes after the beginning of the buffer, which is cache aligned.
As far as I know, the headroom does not contain any data. So I wonder why there should be a headroom in the mbuf？In addition, the default size of HEADROOM is 128B.I've tried setting the HEADROOM size to something bigger, like 5120B, and it doesn't affect performance. So I want to know what is the purpose of the existence of the Headroom, and is there any theoretical limit to the size of the Headroom?
A DPDK mbuf's headroom is suited for prepending extra data (typically, metadata) to the packet data. Its use cases are PMD-specific. For example, virtio PMD stores struct virtio_net_hdr in it. Other drivers may use this for offload results: checksum validity status, packet classification, hash values, you name it.
As for the upper limit, it depends on the mbuf object's maximum size and on the minimum packet data room, which, in turn, might be PMD-specific. For the majority of use cases, it's OK to just rely on default size.
Related
I have just started learning about Blockchain and I am still new to most of the concepts. So please help me out with this. I have read in my class notes that SHA-256 algorithm has a block size of 512 bits, and Bitcoin uses this algorithm too. But when I researched online it says that Bitcoin has a block size of 1MB.
What am I missing over here?
So the concept of "Block" is different in the context of a hash function and the Bitcoin blockchain.
In the context of hash functions, SHA256 processes input data in chunks, and the size limit of each data chunk is called block size. In the case of SHA256, the block/data chunk size limit is 512bits.
In the context of Bitcoin, there's a data structure called "block". Each block contains metadata info and a group of transactions. The hash of block i is packed into the metadata of block i+1, which constructs a connection between these 2 blocks, thus it's called Blockchain. It's like a chain of blocks with each block containing the hash of its previous block. The size limit of each block, in the context of Bitcoin, is 1MB. It limits how many transactions can be packed into a single block because the size of the block matters for the decentralization of the blockchain. Lower the size of the block, lower the requirement for the network nodes' bandwidth and thus more nodes can join the network.
If you want to learn more comprehensively about Bitcoin, this is the go to material I would recommend: https://github.com/bitcoinbook/bitcoinbook
I am planning to create id CHARACTER VARYING(100) ENCODE ZSTD,
id2 CHARACTER VARYING(5000) ENCODE ZSTD.
Now my id and id1 are about size 20 characters only.
In AWS Redshift does the space allocation happen based on actual data size which is 20 or does it allocate first based on defined size which is 100 and 5000 respectively. If so how is the performance effected on these scenarios.
Thanks
TOM
Two things here.
Storage: With varchars, the amount of space consumed is based on the actual amount of space required, not the length (in bytes) declared.
Query Performance: Redshift does not know in advance how many bytes will be required to hold the varchar. It allocates the number of bytes based on the length declared for the varchar. It will cause queries to consume more memory which can in certain cases cause queries to spill to disk. This can have particularly negative impact on vacuum performance.
Summary: Declare varchars to be as short as possible. So, in your case if it's 20 or so, maybe 25-30 would be a good length to go with.
Amazon Redshift stores data using a Compression Encoding, so it is not vital to allocate the minimum space.
It is often best to allow Redshift to choose the compression type when data is loaded via the COPY command, rather than specifying it yourself. This will result in the most efficient method being chosen, based on the first 100,000 rows loaded.
I am repeatedly sending data over UDP socket. Data is of format.
int32_t bid
int32_t ask
int32_t price
All fields are conditional. so, I am using 3 additional bits to track which fields are present in the message. And I am writing these fields to char array before sending.
So my format becomes.
[ first 3 bits | 5 bits] [ ? 32 ] [ ? 32 ] [ ? 32 ]
Problem is that I am wasting 5 bits. I can do dirty pointer airthmetic with binary operations to save those bits but it may be taking away the processing speed. How can I do it cleanly and efficiently in c++ ?
Please provide a simple code snippet for this.
If you care so much about those 5 bits, then you very probably can save much more by dynamically reducing the size of the fields for the bid, ask and price fields. Then in your header field you can allocate two bits per each of your payload fields for holding three possible values:
0 - the field is not present
1 - the field is present and is encoded in 16 bits
2 - the field is present and is encoded in 32 bits
I think the devil is in the detail in these things. It might be wise to study the statistics of the message you send for it might be possible to devise a scheme that improves the average performance.
For example if you are sending a bunch of these at a time (so that the comments about the size of the headers could be misplaced) then you could arrange the message block to be eight arrays (each with a count): first the messages with all 3 fields, then the messages with just, say bid and ask and so on. This does add 8 counts to the message, but means you don't send fields that aren't there; whether you save on average will be depend on the size of the blocks and the statistics of the messages. If some of the combinations were rare you could have a type field before the arrays that specifies which types are there.
Another thing to consider is whether you could steal some bits from the fields. For example, if you had, say, the bid field, do you need the full 32 bits for the others? Could, say, the ask be encoded as a 30 bit difference from the bid?
As already discussed in comments, the first thing is to combine multiple messages into a single UDP packet. And it's not so simple as it can look. The biggest challenge here is to decide the packet size.
The max UDP payload size is 65,507 bytes (I assume it's UDP over IPv4 by default). If UDP payload size is bigger than MTU size UDP will silently segment the packet. In real life UDP packets size usually is equal or less than MTU size. But even MTU size (~1500 bytes) can be too big. There was some research for multimedia streaming application that stated that large UDP packets were more often dropped if network was congested and recommended to use something like 400 bytes payload size as a good balance between a chance to be dropped and to not waste bandwidth for UDP/IP headers. Again, it depends on your application, mainly on your data traffic.
Then, you can apply different compression techniques. bid can be compressed by e.g. Variable Length Encoder or Run Length Encoder, depends on the nature of your bid.
I've no idea what ask field is, but price looks like a good candidate for a fixed point number. If ask is related to price maybe it's worth to send the difference between them and saving some bits.
First of all, decide how many bits per field you really need, then decide how to arrange them to minimise gaps or to optimise performance. Bit manipulation is quite expensive, minimising data copying also can optimise performance.
I have a steady stream of timestamped data, of which I want to always keep the last 5 seconds of data in a buffer.
Furthermore, I would like to provide support for extracting data of a given subinterval of the 5 seconds, so something like
interval = buffer.extractData(startTime, endTime);
What std data structure would be most appropriate for this?
1) The fact that a new sample pushes an old sample out hints that a Queue would be a good data structure
2) The fact that we have to have random access to any elements, in order to obtain the sub interval maybe suggests that vector is appropriate.
Furthermore, what would be a good way to present the subinterval to the user?
My suggestion would be using two iterators?
Unless you are in a fairly performance critical part of the code, a deque would seem reasonable. It can grow and shrink to accommodate changes in your data rate and has reasonable performance for double-ended queue operations and random access.
If the code is performance sensitive (or, even worse, has real-time requirements on top, as is the case with many timestamped buffers), you need to prevent memory allocations as much as possible. You would do this by using a ring buffer with a preallocated array (be it through unique_ptr<T[]> or vector) and either dropping elements when the buffer size is exceeded, or (taking one for the team and) increasing its size.
By never reducing size again, your ring buffer might waste some memory, but remember that in most cases memory is fairly plentiful.
Representing intervals by two iterators or a range object is both common, and although the C++ standard library often prefers iterators, my personal preference is for range objects due to their (in my opinion) slightly better usability.
How to optimize reading and writing speed on std::ifstream and std::ofstream in standard C++/C++11 (no POSIX functions) ? (1 <- as there is several questions, these numbers identify the different points)
I don't know exactly the role of buffers so can you confirm that :
for reading : a large part of the file is preloaded in memory (so the buffer size define the size of this large part) (2)
for writing : data are written in memory and once the buffer is full, it is transfered from memory to the file system (3)
How to set the buffer size on std::ifstream and std::ofstream ? (4)
Considering that I work with very large binary files (several 10 GB), and that filesystems are oftenly optimal for large reading/writting, can I define a buffer size of like 100 MB ? If it will decrease performance, why ? (5)
Finally, is the default buffer "smart" in the sense that ifstream/ofstream will detect how much data you are reading/writing to the file and adapt the buffer size to provide a maximum speed ? (6)
Your description of how the buffering works is correct, AFAICS.
A buffer size larger than 1 MB is unlikely to buy you anything, though. In reality, the sweet spot is likely far below that value. Note that the buffers used by std::ifstream and std::ofstream have nothing to do with the disk cache - that's the kernel's job, and it does it at its own discretion. The stream buffer only affect the maximum number of bytes transferred to or from the kernel with one syscall. Thus, the ideal buffer size doesn't depend on how much data you transfer. What is does depend on is
The overhead cost of syscalls. Higher overhead means you'll want to transfer more data in one go.
The overhead cost of buffer management. Probably larger for larger buffers, if anything.
CPU cache trashing effects. Will strongly favour smaller buffers.
Since (1) works in favour of larger buffers while (2) and (3) work in favour of smaller buffers, there will be a sweet spot somewhere. Since the CPU cache size is likely to be a couple of megabytes or so, a buffer size which approaches that limit will see a severe effect from (3), so the sweet spot surely lies beneath 1 MB or so. You can probably ignore (2), so it remains to estimate (1) to get a lower bound on the buffer size. Assume that a syscall costs somewhere around 1000 cycles or so, and assume that raw copying speed of CPU+memory is 4 bytes/cycle. Transferring 4k then costs about as much as doing one syscall. So with a buffer size of 20k, syscall overhead is roughly 20%, and at a buffer size of 100k it's about 4%. he ideal buffer size is thus somewhere in the range of a couple of hundred kB independent of the file size!.
You can probably trust your standard library implementation to get that right, unless profiling provides you with hard evidence that there is a buffering issue which affects performance.