Embedding cache aligned meta data inside mbuf - dpdk

Question

I'm developing my own dpdk application and I wish received packets to go through several threads in series. Each individual thread has it's own duty of inspecting packets and generating some metadata for each individual packet. It appears to be the easiest and most efficient way to transfer packets between threads is using rte rings. However I need to transfer the metadata generated by each thread to the next thread as well. I have tried doing this using array of structures for metadata and parsing a pointer to next thread. However this method proved to be inefficient since I got lot of cache misses.
As a solution I came up with idea of putting metadata generated by each thread into mbufs themselves. It seems to be doable with "Dynamic fields" of mbufs. However documentation of this feature seems to be very limited. For my application I wish to use a metadata field inside dynamic field something like this,
typedef struct {
uint32_t packet_id;
uint64_t time_stamp;
uint8_t ip_v;
uint32_t length;
.........
.........
} my_metadata_field;
What I don't understand is how much space I can use for dynamic field? The only thing mentioned about this on dpdk documentation is,
"10.6.1. Dynamic fields and flags
The size of the mbuf is constrained and limited; while the amount of
metadata to save for each packet is quite unlimited. The most basic
networking information already find their place in the existing mbuf
fields and flags.
If new features need to be added, the new fields and flags should fit
in the “dynamic space”, by registering some room in the mbuf
structure:
dynamic field -
named area in the mbuf structure, with a given size (at least 1 byte) and alignment constraint."
which doesn't make much sense for me. How much memory I have for this field? If it's almost unlimited, what are the tradeoffs I have to deal with if I use a large metadata field? (performance wise)
I use dpdk 20.08
Edit:
After some digging I have abandoned the idea of using dynamic field for metadata since lack of documentation and it doesn't appears to be able to hold more than 64bits.
I am looking for an easy way to embed my metadata inside cache aligned mbufs (preferably using a struct like above) so I can use rte rings to share them between threads. I'm looking for any documentation or reference project for me to begin with.

Answer 1

There are a couple of ways to carry metadata along with MBUF. Following are the options to do the same
in function rte_mempool_create instead of passing private_data_size as 0 pass the size as custom metadata size.
in function rte_pktmbuf_pool_create instead of passing priv_size as 0 pass the size as custom metadata size
if size of metadata is less than 128 Bytes, use typecast to access memory area right after rte_mbuf
If there are no external buffer used in DPDK application, update rte_mbuf shinfo or next
Solution 1: rte_mempool_create("FIPS_SESS_PRIV_MEMPOOL", 16, sess_sz, 0, sizeof(my_metadata_field), NULL, NULL, NULL, NULL, rte_socket_id(), 0);
Solution 2: rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * nb_ports, MBUF_CACHE_SIZE, sizeof(my_metadata_field), RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
Solution 3:
struct rte_mbuf *bufs[BURST_SIZE];
const uint16_t nb_rx = rte_eth_rx_burst(port, 0, bufs, BURST_SIZE);
if (unlikely(nb_rx == 0))
continue;
for (int index = 0; index < nb_rx; index++)
{
assert(sizeof(my_metadata_field) <= RTE_CACHE_LINE_SIZE);
my_metadata_field *ptr = bufs[index] + 1;
...
...
...
}
Solution 4:
privdata_ptr = rte_mempool_create("METADATA_POOL", 16 * 1024, sizeof(my_metadata_field), 0, 0,
NULL, NULL, NULL, NULL, rte_socket_id(), 0);
struct rte_mbuf *bufs[BURST_SIZE];
const uint16_t nb_rx = rte_eth_rx_burst(port, 0, bufs, BURST_SIZE);
if (unlikely(nb_rx == 0))
continue;
for (int index = 0; index < nb_rx; index++)
{
void *msg = NULL;
if (0 == rte_mempool_get(privdata_ptr, &msg))
{
assert(msg != NULL);
bufs[index]->shinfo = msg;
continue;
}
/* free the mbuf as we are not able to retrieve the private data */
}
/* before transmit or pkt free ensure to release object back to mempool via rte_mempool_put */