built dpdk 19.11.12 from source and i have written an ext app built using these libraries. I use dpdk-devbind to bind a device to igb_uio .
The device i have is an Intel X550. When i run testpmd it seems to recognize the device but int my app rte_eth_dev_count_avail always returns 0
I would belive this has to either do with the way i have built my dpdk library or my app
this is extract of the Makefile i am using to build my app
RTE_SDK ?= /usr/local/share/dpdk
RTE_TARGET ?= x86_64-default-linuxapp-gcc
include $(RTE_SDK)/mk/rte.vars.mk
# binary name
APP = myapp
# all source are stored in SRCS-y
SRCS-y := main.c
WERROR_FLAGS += -Werror
CFLAGS += -O3 -g
CFLAGS += $(WERROR_FLAGS) -Wno-address-of-packed-member
CFLAGS += -Wall
CFLAGS += -pie -fPIE -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector-strong
CFLAGS += -DSTATSD
LDFLAGS += -z relro -z now
LDFLAGS += -ljansson -lrte_pmd_ixgbe
LDLIBS += -lrte_pmd_ixgbe
# disable since libxenstore.so isn't linked properly
#CONFIG_RTE_LIBRTE_PMD_XENVIRT = n
CONFIG_RTE_LIBRTE_PMD_IXGBE = y
include $(RTE_SDK)/mk/rte.extapp.mk
I build my dpdk libraries using this
RUN make -C /dpdk/dpdk-stable-${DPDK_VERSION}/ config T=x86_64-native-linuxapp-gcc
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_BUILD_SHARED_LIB=n/CONFIG_RTE_BUILD_SHARED_LIB=y/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=n/CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=y/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LIBRTE_E1000_DEBUG_RX=n/CONFIG_RTE_LIBRTE_E1000_DEBUG_RX=y/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LIBRTE_E1000_DEBUG_TX=n/CONFIG_RTE_LIBRTE_E1000_DEBUG_TX=y/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX=n/CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX=y/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX=n/CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX=y/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LOG_DP_LEVEL=RTE_LOG_INFO/CONFIG_RTE_LOG_DP_LEVEL=RTE_LOG_DEBUG/g > /tmp/out && mv /tmp/out build/.config
RUN cd /dpdk/dpdk-stable-${DPDK_VERSION}/ && cat build/.config | sed -e s/CONFIG_RTE_LOG_HISTORY=256/CONFIG_RTE_LOG_HISTORY=2048/g > /tmp/out && mv /tmp/out build/.config
i can bind to the interfaces using dpdk-devbind and testpmd also seems to identify and be able to bind to these devices
but my app is not able to recognize the devices.
What am i missing here ?
these are the logs --log-level=8 for EAL logs and I get this output
Sep 09 04:56:50 {"timestamp": 1662717410.8821511, "level": "info", "message": "Initialising EAL ..."}
Sep 09 04:56:50 EAL: Detected 64 lcore(s)
Sep 09 04:56:50 EAL: Detected 2 NUMA nodes
Sep 09 04:56:50 EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
Sep 09 04:56:50 EAL: Selected IOVA mode 'PA'
Sep 09 04:56:50 EAL: No available hugepages reported in hugepages-1048576kB
Sep 09 04:56:50 EAL: Debug dataplane logs available - lower performance
Sep 09 04:56:50 EAL: Debug dataplane logs available - lower performance
Sep 09 04:56:50 EAL: Probing VFIO support...
Sep 09 04:56:50 EAL: VFIO support initialized
Sep 09 04:56:50 EAL: Probing VFIO support...
Sep 09 04:56:50 EAL: VFIO support initialized
also adding the -d parameter with the directory where the plugins are located /usr/local/lib/ gives me an error where the plugin is not found
the eal param initialized is
--master-lcore 4 -l 4,5,6,7 --log-level 8 -d /usr/local/lib/
Sep 09 04:47:29 EAL: Detected 64 lcore(s)
Sep 09 04:47:29 EAL: Detected 2 NUMA nodes
Sep 09 04:47:29 EAL: librte_compressdev.so.0.200: cannot open shared object file: No such file or directory
Sep 09 04:47:29 EAL: FATAL: Cannot init plugins
Sep 09 04:47:29 EAL: Cannot init plugins
the librte_compressdev.so.0.200 is present in the path
ls -l /usr/local/lib/librte_compressdev.so.0.200
-rwxr-xr-x 1 root root 38024 Sep 8 23:51 /usr/local/lib/librte_compressdev.so.0.200
[based on the live debug]
DPDK 17.11 LTS use libdpdk.so and a complete shared library, which makes all the necessary ethdev PMD available. hence using the current makefile, it pans out properly without any issues.
But moving to DPDK 19.11 LTS, libdpdk.so is no longer availble. hence one have to manually link either in shared or static libraries properly (as pointed out in comments). For example
DPDK_CFLAGS=" -DHAVE_DPDK -I${RTE_SDK}/${RTE_TARGET}/include/ -include ${RTE_SDK}/${RTE_TARGET}/include/rte_config.h "
DPDK_LDFLAGS="-I$RTE_SDK/lib/librte_eal/linux/eal/include \
-DRTE_USE_FUNCTION_VERSIONING \
-I$RTE_SDK/$RTE_TARGET/include -include $RTE_SDK/$RTE_TARGET/include/rte_config.h \
-D_GNU_SOURCE -W -Wall -DALLOW_EXPERIMENTAL_API \
-L$RTE_SDK/$RTE_TARGET/lib -Wl,-lrte_flow_classify -Wl,--whole-archive -Wl,-lrte_pipeline -Wl,--no-whole-archive \
-Wl,--whole-archive -Wl,-lrte_table -Wl,--no-whole-archive -Wl,--whole-archive \
-Wl,-lrte_port -Wl,--no-whole-archive -Wl,-lrte_pdump -Wl,-lrte_ip_frag -Wl,-lrte_meter \
-Wl,-lrte_fib -Wl,-lrte_rib -Wl,-lrte_lpm -Wl,-lrte_acl -Wl,-lrte_power -Wl,-lrte_kni \
-Wl,-lrte_efd -Wl,-lrte_bpf -Wl,-lrte_ipsec -Wl,--whole-archive -Wl,-lrte_cfgfile -Wl,-lrte_gro \
-Wl,-lrte_gso -Wl,-lrte_hash -Wl,-lrte_member -Wl,-lrte_vhost -Wl,-lrte_kvargs -Wl,-lrte_mbuf \
-Wl,-lrte_net -Wl,-lrte_ethdev -Wl,-lrte_cryptodev -Wl,-lrte_security -Wl,-lrte_compressdev \
-Wl,-lrte_eventdev -Wl,-lrte_timer -Wl,-lrte_mempool -Wl,-lrte_stack -Wl,-lrte_mempool_ring \
-Wl,-lrte_ring -Wl,-lrte_pci -Wl,-lrte_eal -Wl,-lrte_cmdline -Wl,-lrte_reorder -Wl,-lrte_sched \
-Wl,-lrte_rcu -Wl,-lrte_bus_pci -Wl,-lrte_bus_vdev -Wl,-lrte_mempool_bucket -Wl,-lrte_mempool_stack \
-Wl,-lrte_pmd_af_packet \
-Wl,-lrte_pmd_ixgbe \
-Wl,-lrte_pmd_memif \
-Wl,-lrte_pmd_null \
-Wl,-lpcap -Wl,-lrte_pmd_ring -Wl,-lrte_pmd_tap -Wl,-lrte_pmd_virtio -Wl,-lrte_bus_vmbus \
-Wl,-lcrypto -Wl,-lrte_pmd_null_crypto -Wl,-lrte_pmd_crypto_scheduler -Wl,-lrte_pmd_virtio_crypto \
-Wl,-lrte_pmd_skeleton_event -Wl,-lrte_pmd_sw_event \
-Wl,--no-whole-archive -Wl,-lrt -Wl,-lm -Wl,-lnuma -Wl,-ldl -Wl,-export-dynamic -Wl,-export-dynamic \
-L$RTE_SDK/examples/l2fwd/build/lib -L$RTE_SDK/$RTE_TARGET/lib -Wl,--as-needed"
Since DPDk 19.11 LTS makes uses on meson-ninja, my suggestion and recommendation is to make use of pkg-config to built the right CFLAGS and LDFLAGS. Please refer StackOverflow query on using dpdk cmake too
Note: recommended to build a custom target folder rather than make install, since the built and application run is done in docker.
I did a static linking of libdpdk to my application and when I start the application EAL init fails.
Logs:
EAL: Detected 80 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
MBUF: error setting mempool handler
EAL: Error - exiting with code: 1
Cause: Cannot init packet mbuf pool Invalid argument
I installed DPDK-20.11 in my test server, using the following commands.
# meson build
#cd build
#ninja ; ninja install
I built my application Makefile taking l2fwd Makefile as an example, not sure what I am missing.
Looking out for suggestions.
[Edit-1]
int main(int argc, char **argv)
{
int ret; unsigned lcore_id;
ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_panic("Cannot init EAL\n");
struct rte_mempool *mp =
rte_pktmbuf_pool_create("packet_pool", 8192, 64, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
return 0;
}
How to build
copy makefile from l2fwd to custom app folder
execute make static
run the application sudo ./a.out
[EDIT-1] update from live debug
Target OS: CENTOS 7
DPDK version: 20.11.1
There are multiple reasons which were causing error
Installing Mellanox libverbs affects the cflags and ldflags for dpdk libdpdk.pc
DPDK 21.11.1 meson build (meson -Dexamples=l2fwd build; ninja -C install) does not match l2fwd Makefile option of make static
using pkg-config from the installed target was also incorrect.
To fix the error we edited /usr/local/lib64/pkgconfig/libdpdk.pc to include the static libraries under -Wl,--whole-archive and -Wl,--no-whole-archive
it works
#include <rte_cycles.h>
#include <rte_prefetch.h>
#include <rte_lcore.h>
#include <rte_per_lcore.h>
#include <rte_branch_prediction.h>
#include <rte_interrupts.h>
#include <rte_random.h>
#include <rte_debug.h>
#include <rte_ether.h>
#include <rte_ethdev.h>
#include <rte_mempool.h>
#include <rte_mbuf.h>
#include <rte_string_fns.h>
int main(int argc, char **argv)
{
int ret; unsigned lcore_id;
ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_panic("Cannot init EAL\n");
struct rte_mempool *mp =
rte_pktmbuf_pool_create("packet_pool", 8192, 64, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
if (mp == NULL)
rte_panic("Cannot init EAL\n");
printf("done!!!!!!!!!!!!");
return 0;
}
build: sudo make static
Output
$ sudo ./build/l2fwd
EAL: Detected 32 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
done!!!!!!!!!!!!
I am trying to build a library on my mac (OSX Version 10.14.1, Xcode Version 10.1) and it fails giving following error:
Building target: application.a
ar -r "application.a" api.o
ar: creating archive application.a
/opt/local/bin/ranlib: object: application.a(api.o) malformed object (unknown load command 1)
ar: internal ranlib command failed
make[1]: *** [application.a] Error 1
make: *** [all] Error 2
by running 'otool -l lib/api.o' I get the following result:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777223 3 0x00 1 5 616 0x00002000
Load command 0
cmd LC_SEGMENT_64
cmdsize 472
segname
vmaddr 0x0000000000000000
vmsize 0x0000000000004fb0
fileoff 648
filesize 20400
maxprot 0x00000007
initprot 0x00000007
nsects 5
flags 0x0
Section
sectname __text
segname __TEXT
addr 0x0000000000000000
size 0x00000000000040ae
offset 648
align 2^4 (16)
reloff 21048
nreloc 329
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __cstring
segname __TEXT
addr 0x00000000000040ae
size 0x00000000000000cb
offset 17206
align 2^0 (1)
reloff 0
nreloc 0
flags 0x00000002
reserved1 0
reserved2 0
Section
sectname __const
segname __TEXT
addr 0x0000000000004180
size 0x000000000000003f
offset 17416
align 2^4 (16)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __compact_unwind
segname __LD
addr 0x00000000000041c0
size 0x00000000000005e0
offset 17480
align 2^3 (8)
reloff 23680
nreloc 47
flags 0x02000000
reserved1 0
reserved2 0
Section
sectname __eh_frame
segname __TEXT
addr 0x00000000000047a0
size 0x0000000000000810
offset 18984
align 2^3 (8)
reloff 0
nreloc 0
flags 0x6800000b
reserved1 0
reserved2 0
Load command 1
cmd ?(0x00000032)
cmdsize 24
Load command 2
cmd LC_DATA_IN_CODE
cmdsize 16
dataoff 24056
datasize 8
Load command 3
cmd LC_SYMTAB
cmdsize 24
symoff 24064
nsyms 93
stroff 25552
strsize 1668
Load command 4
cmd LC_DYSYMTAB
cmdsize 80
ilocalsym 0
nlocalsym 15
iextdefsym 15
nextdefsym 39
iundefsym 54
nundefsym 39
tocoff 0
ntoc 0
modtaboff 0
nmodtab 0
extrefsymoff 0
nextrefsyms 0
indirectsymoff 0
nindirectsyms 0
extreloff 0
nextrel 0
locreloff 0
nlocrel 0
I am guessing load command miss matches between my gcc and osx version, but I can't find 0x00000032 command on mach-o/loader.h. does anyone have a clue about this?
here is the 'otool --version':
llvm-otool(1): Apple Inc. version cctools-895
LLVM (http://llvm.org/):
LLVM version 4.0.1
Optimized build.
Default target: x86_64-apple-darwin18.2.0
Host CPU: haswell
Registered Targets:
aarch64 - AArch64 (little endian)
aarch64_be - AArch64 (big endian)
amdgcn - AMD GCN GPUs
arm - ARM
arm64 - ARM64 (little endian)
armeb - ARM (big endian)
bpf - BPF (host endian)
bpfeb - BPF (big endian)
bpfel - BPF (little endian)
hexagon - Hexagon
lanai - Lanai
mips - Mips
mips64 - Mips64 [experimental]
mips64el - Mips64el [experimental]
mipsel - Mipsel
msp430 - MSP430 [experimental]
nvptx - NVIDIA PTX 32-bit
nvptx64 - NVIDIA PTX 64-bit
ppc32 - PowerPC 32
ppc64 - PowerPC 64
ppc64le - PowerPC 64 LE
r600 - AMD GPUs HD2XXX-HD6XXX
riscv32 - 32-bit RISC-V
riscv64 - 64-bit RISC-V
sparc - Sparc
sparcel - Sparc LE
sparcv9 - Sparc V9
systemz - SystemZ
thumb - Thumb
thumbeb - Thumb (big endian)
x86 - 32-bit X86: Pentium-Pro and above
x86-64 - 64-bit X86: EM64T and AMD64
xcore - XCore
Do you have macports installed? I had the same issue and it seems that macports cctool package install /opt/local/bin/otool which is likely ahead in your path. I suspect that /usr/bin/otool gives the right one ...
First run ./configure it worked,
then make -j4, it will report
options.c:11145: error: narrowing conversion of '-0x00000000000000001' from 'int' to 'short unsigned int' inside { }
GMP、MPFR、MPC install successed
cat /proc/version
Linux version 2.6.32-504.23.4.el6.x86_64 (mockbuild#x86-029.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4-9) (GCC) ) #1 SMP Fri May 29 10:16:43 EDT 2015
file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.6 (Santiago)
Release: 6.6
Codename: Santiago
I compile the following C++ code
// main.cpp
#include <cstdio>
int main() {
unsigned char tab[4] = {0};
printf("%d\n", __builtin_popcount(*((int *)tab)));
}
using command line:
g++ -o prog main.cpp -mpopcnt
When I run the program I get error:
Illegal instruction
Compiling without -mpopcnt does not give an error (it just prints 0).
Question: what is causing this error?
I am compiling and running the program on the same machine.
Valgrind detects no problem. Running
valgrind --leak-check=full ./prog
gives
==12917== Memcheck, a memory error detector
==12917== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==12917== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==12917== Command: ./prog
==12917==
0
==12917==
==12917== HEAP SUMMARY:
==12917== in use at exit: 0 bytes in 0 blocks
==12917== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==12917==
==12917== All heap blocks were freed -- no leaks are possible
==12917==
==12917== For counts of detected and suppressed errors, rerun with: -v
==12917== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
Below I give some specifications of my system.
I'm using Ubuntu 12.04. Running
uname -a
gives me
Linux wtu-82 3.2.0-65-generic #99-Ubuntu SMP Fri Jul 4 21:03:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Running
g++ -v
gives
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.4-1ubuntu1~12.04' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.6.4 (Ubuntu/Linaro 4.6.4-1ubuntu1~12.04)
The output of
cat /proc/cpuinfo
is
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU E8500 # 3.16GHz
stepping : 10
microcode : 0xa0c
cpu MHz : 2000.000
cache size : 6144 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
bogomips : 6317.48
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU E8500 # 3.16GHz
stepping : 10
microcode : 0xa0c
cpu MHz : 2000.000
cache size : 6144 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow vnmi flexpriority
bogomips : 6317.38
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
POPCNT was introduced in SSE 4.2. Your processor is SSE 4.1. So, the instruction is simply missing. You get an illegal instruction error when you force the compiler, with -mpopcnt, to generate code using an instruction your processor doesn't know about.