DPDK occurs "failed to allocate private data" in rte_eal_init-a possibly unknown dpdk bug? - dpdk

Question

When I searched for this problem, I found that almost no one else encountered such a problem.It's strange.By the way, I use dpdk-19.11.12 and Ubuntu-20.04
After looking at the dpdk source code, I know that the above code is in the rte_eth_dev_create function:
if (priv_data_size) {
ethdev->data->dev_private = **rte_zmalloc_socket**(
name, priv_data_size, RTE_CACHE_LINE_SIZE,
device->numa_node);
if (!ethdev->data->dev_private) {
RTE_LOG(ERR, EAL, "failed to allocate private data");
retval = -ENOMEM;
goto probe_failed;
}
}
It seems rte_zmalloc_socket return a NULL pointer. Why this happens?I allocated the relevant hugepage memory as requested.
Some Information：
EAL: Detected 4 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:31:00.0 on NUMA socket 3
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:31:00.1 on NUMA socket 3
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:51:00.0 on NUMA socket 5
EAL: probe driver: 8086:10fb net_ixgbe
failed to allocate private data
EAL: Requested device 0000:51:00.0 cannot be used
EAL: PCI device 0000:51:00.1 on NUMA socket 5
EAL: probe driver: 8086:10fb net_ixgbe
failed to allocate private data
EAL: Requested device 0000:51:00.1 cannot be used
Hugepages： (cat /proc/meminfo | grep Huge)
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 20
HugePages_Free: 19
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 20971520 kB
and mount information:
(mount | grep huge)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=1024M)
nodev on /mnt/huge type hugetlbfs (rw,relatime,pagesize=1024M)
NIC:
Network devices using DPDK-compatible driver
============================================
0000:51:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe,vfio-pci
0000:51:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe,vfio-pci
Network devices using kernel driver
===================================
0000:31:00.0 'I350 Gigabit Network Connection 1521' if=enp49s0f0 drv=igb unused=igb_uio,vfio-pci
0000:31:00.1 'I350 Gigabit Network Connection 1521' if=enp49s0f1 drv=igb unused=igb_uio,vfio-pci *Active*
Numa information:(numactl -H)
node 5 cpus: 30 31 32 33 34 35 78 79 80 81 82 83
node 5 size: 64475 MB
node 5 free: 58743 MB
82599ES NICs are on numa node 5.
I thought I did all the initialization but rte_eal_init returned the error "failed to allocate private data"
Any ideas on this issue? Thanks for help.
-----------------------update-----------------------------------------
Since this problem occurs in rte_zmalloc_socket(),I guess that there is something wrong with my configuration in Hugepages.But as posted above,i check Hugepages with command:
1.cat /proc/meminfo | grep Huge
2.mount | grep huge
first command checks available 1G Hugepages and second checks mount situation.
Result above seems to be normal which confuses me most.
Any clues?Thanks.

Answer 1

Thanks, the issue has been resolved.
The cause of the problem is that my cpu has 8 numa nodes and nics bound to dpdk are on node 5,but by default dpdk only supports 4 numa nodes.
My experience of finding the problem is like this：
rte_eth_dev_create calls rte_zmalloc_socket to allocate private-data
rte_zmalloc_socket directly calls rte_malloc_socket:
void *
rte_zmalloc_socket(const char *type, size_t size, unsigned align, int socket)
{
void *ptr = rte_malloc_socket(type, size, align, socket);
#ifdef RTE_MALLOC_DEBUG
/*
* If DEBUG is enabled, then freed memory is marked with poison
* value and set to zero on allocation.
* If DEBUG is not enabled then memory is already zeroed.
*/
if (ptr != NULL)
memset(ptr, 0, size);
#endif
return ptr;
}
rte_malloc_socket calls malloc_heap_alloc with socket_arg = 5 in my case:
void *
rte_malloc_socket(const char *type, size_t size, unsigned int align, int socket_arg)
{
/* return NULL if size is 0 or alignment is not power-of-2 */
if (size == 0 || (align && !rte_is_power_of_2(align)))
return NULL;
/* if there are no hugepages and if we are not allocating from an
* external heap, use memory from any socket available. checking for
* socket being external may return -1 in case of invalid socket, but
* that's OK - if there are no hugepages, it doesn't matter.
*/
if (rte_malloc_heap_socket_is_external(socket_arg) != 1 &&
!rte_eal_has_hugepages())
socket_arg = SOCKET_ID_ANY;
return malloc_heap_alloc(type, size, socket_arg, 0,
align == 0 ? 1 : align, 0, false);
}
4.In malloc_heap_alloc:
void*
malloc_heap_alloc(const char *type, size_t size, int socket_arg, unsigned int
flags, size_t align, size_t bound, bool contig)
{
int socket, heap_id, i;
void *ret;
/* return NULL if size is 0 or alignment is not power-of-2 */
if (size == 0 || (align && !rte_is_power_of_2(align)))
return NULL;
if (!rte_eal_has_hugepages() && socket_arg < RTE_MAX_NUMA_NODES)
socket_arg = SOCKET_ID_ANY;
if (socket_arg == SOCKET_ID_ANY)
socket = malloc_get_numa_socket();
else
socket = socket_arg;
/* turn socket ID into heap ID */
**heap_id = malloc_socket_to_heap_id(socket);**
/* if heap id is negative, socket ID was invalid */
**if (heap_id < 0)
return NULL;**
...
}
My program returns NULL because heap_ip = -1 while socket = 5,which eventually causes rte_zamlloc_socket return NULL and failed to allocate private data for nics.
By running " meson configure " in dpdk build directory,I got such result:
max_numa_nodes 4 maximum number of NUMA nodes supported by EAL
So in my case, I need to use either one of the steps to rebuild dpdk.
rebuild from scratch "cd [dpdk parent folder]; rm -rf build; meson -Dmax_numa_nodes=8 build"
to reuse existing build folder "cd [dpdk parent folder]; meson --reconfigure -Dmax_numa_nodes=8 build"
After that, everything is back to normal.
Thanks for help,and hope this can help anyone else why meet the same problem.:)