when another process mmap and read the pci resource file
enter image description here
Env:
Eth X520 ixgbe Driver
DPDK 19.11
OS centos 7
Related
I was trying to get the l2fwd application to work but it keeps showing this error. I dont understand I have the NICs properly bound and hugepages configured
The error
`
./dpdk-l2fwd -l 0-3 -n 1 --no-telemetry -- -q 8 -p ffff
EAL: Detected CPU lcores: 6
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
MAC updating enabled
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - bye
root#dpdku:~/dpdk/build/examples#
`
Hugepages
`
dpdk-hugepages.py -s
Node Pages Size Total
0 600 2Mb 1Gb
Hugepages mounted on /dev/hugepages
`
NICs:
`
dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:00:08.0 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter 8139' drv=uio_pci_generic unused=8139too,vfio-pci
0000:00:09.0 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter 8139' drv=uio_pci_generic unused=8139too,vfio-pci
Network devices using kernel driver
===================================
0000:00:03.0 'RTL-8100/8101L/8139 PCI Fast Ethernet Adapter 8139' if=ens3 drv=8139cp unused=8139too,vfio-pci,uio_pci_generic *Active*
`
DPDK application l2fwd is just doing the right thing, not identifying the NIC you have shared. There is nothing wrong with the behaviour as this is the expected behaviour. Reason for the same is shared below as
The nic RTL-8100 is Realtek Semiconductor RTL-8100/8101L/8139 PCI Fast Ethernet Adapter
Following DPDK device enter link description here does not list the support for Realtek NIC
One can check nic vs features to go through the various PF and VF PMD, realtek driver is absent.
Also checking the online driver net PMD realtek driver is not supported.
Hence the only way to use the Realtek nic dpdk is to
bind the nic back to the kernel
use either PCAP PMD to access the device into DPDK
or use AF_XDP (if RTL supports the same), you can use AF_PACKET PMD.
Note: please at least read up on documentation or hw list from dpdk.org to understand the library and supported PMD.
I have a board with one ethernet interface (eth0) running Linux.
I'm trying to forward all incoming traffic from eth0 to my PMD driver, using dpdk-l2fwd example application.
Here is what I've tried:
./dpdk-l2fwd -c 0x3 --vdev={my_pmd}0 -- -p 0x3 -T 0
I can see that my rx_pkt_burst callback is polled by the application, but that's it.
How can I forward all incoming eth0 packets to my PMD?
I tried to use net_tap, using the following command:
./dpdk-l2fwd -c 0xff --vdev=net_tap0 --vdev={my_pmd}0 -- -p 0x7 -T 0 --portmap="(1,2)"
And my tx_pkt_burst callback is called occasionally, but not when I think it should be called.
For example, if I ping this board from another one, the ping is successful, but the tx_pkt_burst callback is not been called.
I tried to use devbind tool, but no devices are detected:
./usertools/dpdk-devbind.py --status
No 'Network' devices detected
=============================
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
==============================
No 'Misc (rawdev)' devices detected
===================================
No 'Regex' devices detected
===========================
Update
DPDK version - 20.11.
My HW is a embedded device based on NXP's Layerscape.
$ lshw -class network
*-network
description: Ethernet interface
physical id: 3
logical name: eth0
serial: 00:11:22:44:11:44
size: 1Gbit/s
capacity: 1Gbit/s
capabilities: ethernet physical tp mii 10bt-fd 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=fsl_dpaa2_eth driverversion=5.10.35-00002-g3434eea0e1e7-dir duplex=full firmware=7.17 ip=192.168.15.157 link=yes multicast=yes port=twisted pair speed=1Gbit/s
I'm trying to bypass all traffic to the PMD I'm currently developing.
Thanks.
[EDIT-1] clarification of using same interface for DPDK and Kernel routing
Answer> as discussed over comments please refer to DPDKD + kernel on same interface
Based on the information shared there are multiple questions to the single query I'm trying to bypass all traffic to the PMD I'm currently developing. Addressing each one separately below
question 1: using dpdk-l2fwd example application
Answer> DPDK application l2fwd application makes use of basic APi with almost no HW offloads. Based on your environment (I have a board with one ethernet interface (eth0)), the right set of parameters should be -p 0x1 --no-mac-updating -T 1. This will configure the application to receive and transmit packet using single DPDK interface (that is eth0 on your board).
Note: DPDK Application can work with DPDK PMD both physical and virtual
question 2: I tried to use net_tap, using the following command:
Answer> If the intend is to intercept the traffic from physical and then forward to tap interface, then one needs modify the eal arguments as ./build/l2fwd --vdev=net_tap0,iface="my_eth0" -- -p 0x3 -T 1 --no-mac-updating. This will allow the application to probe physical NXP interface (eth0) and make use of Linux TAP interface as secondary interface. Thus any traffic from NXP and TAP will be cross connected such as NXP (eth0) <==> TAP (my_eth0)
question 3: ./usertools/dpdk-devbind.py --status returns empty
Answer> Form the dpdk site supported NIC list NXP dpaa, dpaa2, enetc, enetfec, pfe. Cross checking the kernel driver fsl_dpaa2_eth I think it is safe to assume dpaa2 PMD is supported. As you have mentioned the NIC is not enumerated, it looks like there are certain caveats to such model revision, supported board, BSP package, vendor-sub vendor ID check etc. More details can be found Board Support Package, and DPAA2 NIC guide
Debug & Alternative solutions:
To start with use the Kernel Driver to bring in packets
Use extra logging and debug to identify why the NIC is shown in the application
Approach 1:
Make sure the NIC is bind with kernel driver fsl_dpaa2_eth.
ensure NIC is connected and link is up with ethtool eth0
set to promiscous mode with ifconfig eth0 promisc up
start DPDK application with PCAP PMD, ./build/l2fwd --vdev=net_pcap0,iface=eth0 -- -p 1 --no-mac-updating -T 1
Check packet are received and redirected to PCAP eth0 PMD by checking the statistics.
Approach 2:
Ideally the NIC should be categorized under network device to be probed by debind.py.
Check the device details using lshw -c net -businfo for network.
try checking with lspci -Dvmmnnk [PCIe BUS:Slot:Function id] for network details.
If above details does not show up as network device this might be reason for not getting listed.
Suggestions or workaround: You can try to forcefully bind with igb_uio or vfio-pci (I am not much famialr with NXP SoC) by dpdk-devbind -b vfio-pci [PCIe S:B:F]. Then cross check with lspci -ks [PCIe S:B:F]. Once successfully done, one can start dpdk l2fwd in PMD debug mode with ./build/l2fwd -a [PCIe S:B:F] --log-level=pmd,8 -- -p 1 --no-mac-updating | more. Thus by intercepting and interpreting the logs one can identify what is going
Note:
It is assumed the application is build with static libraries and not dynamic. To build with static libraries use make static for l2fwd.
For the described use case recommended application is basicfwd/skeleton rather than l2fwd.
Found the problem.
I had to unbind eth0 from Linux kernel.
Now I can simply run:
./dpdk-l2fwd -c 0x3 --vdev={MY_PMD}0 -- -p 0x3 -T 1
And all traffic in the physical port is forwarded to my PMD.
Some context first:
I installed and configured OVS-DPDK on VM0 [ubuntu + qemu/kvm + ovs-dpdk]. As a guest running on top of VM0 I have VM1 [ubuntu + dpdk].
After a bit of googling I was able to create vhost-user-client port in OVS and consume it with VM1.
At this point I can see the product of vhost-user-client port by running:
$ sudo dpdk-devbind.py --status
Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=vfio-pci *Active*
0000:07:00.0 'Virtio network device 1041' if=enp7s0 drv=virtio-pci unused=vfio-pci *Active* <--- this is it
However after binding it using command:
sudo dpdk-devbind.py --bind=virtio-pci 0000:07:00.0
When running dpdk-testpmd, I get nb forwarding ports=0
$ sudo dpdk-testpmd
EAL: Detected 3 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:01:00.0 (socket 0)
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP allocation mode: native
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=0
Press enter to exit
I dont see any clues for this in dmesg and not sure where else to look.
How can I debug this issue
As hinted in the comments, DPDK ports will work with uio_pci_generic | vfio-pci | igb_uio. Hence for a virtio network device to be used with DPDK it needs binding with vfio-pci.
Please follow the steps as
ifconfig enp7s0 down
sudo modprobe uio
sudo modprobe vfio-pci (for newer kernel this can be skipped)
sudo echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
sudo dpdk-devbind.py --bind=vfio-pci 0000:07:00.0
One done successfully dpdk application with virtio net can be used.
Note: as mentioned in the comments, it is not clear why virtio net device 0000:07:00.0 is bind with virtio-pci
I am trying to use DPDK 19.11 in centOS 8.0. I have compiled DPDK as per guidelines. Then I did bind ethernet NIC to DPDK driver and left Wifi to Linux kernel using dpdk_setup.sh.
1.When I try to use testpmd the following message is displayed.
"testpmd: No probed ethernet devices" though the port is bounded.
When I try to use dpdk_pdump, I get the message
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - bye
Please find driver allocation
Network devices using DPDK-compatible driver
0000:03:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller 8168' drv=igb_uio unused=r8169
Network devices using kernel driver
0000:02:00.0 'Wireless 3160 08b4' if=wlp2s0 drv=iwlwifi unused=igb_uio Active
Executed testpmd and dpdkpdump from the following folders
/home/vijay/fwldpdk/x86_64-native-linuxapp-gcc/app
testpmd output
$ sudo ./testpmd -c f -n 4 -- -i
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd> show port
Bad arguments
testpmd> show port 0
Bad arguments
testpmd> show port 1
Bad arguments
DPDK_pdump output
$ sudo ./dpdk-pdump
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_75341_87718f3ca82
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - bye
I did refer the following question
DPDK run testpmd failed. EAL:no probed ethernet device.
2.If I put wifi interface also to DPDK driver whether testpmd will forward packets between two DPDK ports?
We bring down NIC interface so that it can be bounded to DPDK driver. Do we need to bring it up once it's attached with DPDK using any command?
Any inputs on how do I resolve it?
The expectation of Having RTL and Wifi are incorrect. List of supported DPDK ports.
I'm new to dpdk and using dpdk-stable-17.11.2 on docker.
OS is Ubuntu 14.04 and the kernel is 3.19.0-80-generic.
I've followed the dpdk-setup.sh as below and bind the igb_uio driver to my device.
[14] x86_64-native-linuxapp-gcc
[17] Insert IGB UIO module
[18] Insert VFIO module
[19] Insert KNI module
[20] Setup hugepage mappings for non-NUMA systems
[21] Setup hugepage mappings for NUMA systems
[22] Display current Ethernet/Crypto device settings
[23] Bind Ethernet/Crypto device to IGB UIO module
[24] Bind Ethernet/Crypto device to VFIO module
[25] Setup VFIO permissions
Network devices using DPDK-compatible driver
============================================
0000:03:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet 168e' drv=igb_uio unused=vfio-pci
Network devices using kernel driver
===================================
0000:03:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet 168e' if=eth1 drv=bnx2x unused=igb_uio,vfio-pci *Active*
0000:04:00.0 'NetXtreme BCM5751 Gigabit Ethernet PCI Express 1677' if=eth0 drv=tg3 unused=igb_uio,vfio-pci *Active*
0000:07:00.0 'NX3031 Multifunction 1/10-Gigabit Server Adapter 0100' if=eth3 drv=netxen_nic unused=igb_uio,vfio-pci
0000:07:00.1 'NX3031 Multifunction 1/10-Gigabit Server Adapter 0100' if=eth4 drv=netxen_nic unused=igb_uio,vfio-pci
I got errors when running testpmd.
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No probed ethernet devices
PANIC in main():
Empty set of forwarding logical cores - check the core mask supplied in the command parameters
5: [x86_64-native-linuxapp-gcc/app/testpmd() [0x46504f]]
4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fe9117c9f45]]
3: [x86_64-native-linuxapp-gcc/app/testpmd(main+0x8fb) [0x460aab]]
2: [x86_64-native-linuxapp-gcc/app/testpmd(__rte_panic+0xb8) [0x458a7a]]
1: [x86_64-native-linuxapp-gcc/app/testpmd(rte_dump_stack+0x1a) [0x4ea2aa]]
What have I missed?
I changed CONFIG_RTE_LIBRTE_BNX2X_PMD=n to CONFIG_RTE_LIBRTE_BNX2X_PMD=y in the $RTE_SDK/config/common_base.
0000:03:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet 168e' drv=igb_uio unused=vfio-pci
[...]
EAL: No probed ethernet devices
This error has solved.
0000:03:00.1 'NetXtreme II BCM57810 10 Gigabit Ethernet 168e' drv=igb_uio unused=vfio-pci
[...]
EAL: No probed ethernet devices
BCM57810 seems to be unsupported by the current version of Broadcom PMD:
https://dpdk.org/doc/guides/nics/bnxt.html
As a workaround you can try to use a supported NIC instead (if possible) or a virtual device (might be much slower).
PANIC in main():
Empty set of forwarding logical cores - check the core mask supplied in the command parameters
You did not list the your command line options, but you definitely should check it for a lcore mask or core list or core map. Here is the list of EAL command line options:
https://dpdk.org/doc/guides/testpmd_app_ug/run_app.html